Instructions to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF",
	filename="Qwen3-4B-Thinking-2507-IQ4_NL.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

Use Docker

docker model run hf.co/magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

Ollama
How to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with Ollama:
```
ollama run hf.co/magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M
```

Unsloth Studio new

How to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF to start chatting

Pi new

How to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with Docker Model Runner:
```
docker model run hf.co/magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M
```

Lemonade

How to use magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull magiccodingman/Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3-4B-Thinking-2507-Unsloth-MagicQuant-Hybrid-GGUF-Q4_K_M

List all available models

lemonade list

MagicQuant GGUF Hybrids - Qwen3 4B Thinking 2507

(DEPRECIATED - Part of MagicQuant v1.0 which had significant flaws. Please utilize v2.0 which is production ready)

MagicQuant is an automated quantization, benchmarking, and evolutionary hybrid-GGUF search system for LLMs.

Each release includes models optimized to outperform standard baseline quants (Q8, Q6, Q5, Q4). If a baseline GGUF exists in this repo, the evolutionary engine couldn’t beat it. If a baseline is missing, it’s because a hybrid configuration outperformed it so completely that including the baseline would've been pointless.

These hybrid GGUFs are built to be as small, fast, and low-drift as possible while preserving model capability.

To dive deeper into how MagicQuant works, see the main repo: MagicQuant on GitHub (by MagicCodingMan)

Notes:

The HuggingFace hardware compatibility where it shows the bits is usually wrong. It doesn't understand hybrid mixes, so don't trust it.
Naming scheme can be found on the MagicQuant Wiki.
(tips) Less precision loss means less brain damage. More TPS means faster! Smaller is always better right?

Precision Loss Guide

0–0.1% → God-tier, scientifically exact
0.1–1% → True near-lossless, agent-ready
1–3% → Minimal loss, great for personal use
3–5% → Borderline, but still functional
5%+ → Toys, not tools, outside MagicQuant’s scope

Learn more about precision loss here.

Table - File Size + TPS + Avg Precision Loss

model_name	file_size_gb	bench_tps	avg_prec_loss
mxfp4_moe-O-Q6K-EQKUD-Q8_0	3.90	369.09	0.0989%
mxfp4_moe-Q-Q5K-EKOUD-Q6K	3.03	394.06	0.1278%
iq4_nl-EQKOUD-Q6K	3.08	413.99	0.1740%
mxfp4_moe-QK-IQ4NL-O-MXFP4-EUD-Q6K	2.84	430.23	0.3832%
Q5_K	2.69	375.72	0.5973%
Q4_K_M	2.33	366.54	1.6668%
mxfp4_moe-QKU-IQ4NL-O-MXFP4-D-Q5K-E-Q6K	2.30	412.13	2.2740%
IQ4_NL	2.23	450.75	2.4657%
mxfp4_moe-EQOU-IQ4NL-KD-Q6K	2.37	472.25	2.5049%

Table - PPL Columns

model_name	gen	gen_er	code	code_er	math	math_er
mxfp4_moe-O-Q6K-EQKUD-Q8_0	10.0081	0.2450	1.5936	0.0128	6.9001	0.1413
mxfp4_moe-Q-Q5K-EKOUD-Q6K	9.9957	0.2441	1.5922	0.0127	6.9036	0.1412
iq4_nl-EQKOUD-Q6K	9.9687	0.2431	1.5927	0.0127	6.8924	0.1409
mxfp4_moe-QK-IQ4NL-O-MXFP4-EUD-Q6K	10.0858	0.2460	1.5949	0.0126	6.9032	0.1403
Q5_K	10.0993	0.2473	1.5978	0.0128	6.9256	0.1413
Q4_K_M	10.3239	0.2536	1.6093	0.0129	6.9423	0.1412
mxfp4_moe-QKU-IQ4NL-O-MXFP4-D-Q5K-E-Q6K	10.4164	0.2569	1.6143	0.0130	6.9825	0.1423
IQ4_NL	10.3718	0.2548	1.6125	0.0129	7.0606	0.1452
mxfp4_moe-EQOU-IQ4NL-KD-Q6K	10.3780	0.2547	1.6178	0.0132	7.0415	0.1443

Table - Precision Loss Columns

model_name	loss_general	loss_code	loss_math
mxfp4_moe-O-Q6K-EQKUD-Q8_0	0.0250	0.1194	0.1524
mxfp4_moe-Q-Q5K-EKOUD-Q6K	0.1488	0.0314	0.2032
iq4_nl-EQKOUD-Q6K	0.4186	0.0628	0.0406
mxfp4_moe-QK-IQ4NL-O-MXFP4-EUD-Q6K	0.7512	0.2010	0.1974
Q5_K	0.8861	0.3832	0.5225
Q4_K_M	3.1297	1.1057	0.7649
mxfp4_moe-QKU-IQ4NL-O-MXFP4-D-Q5K-E-Q6K	4.0537	1.4199	1.3484
IQ4_NL	3.6082	1.3068	2.4820
mxfp4_moe-EQOU-IQ4NL-KD-Q6K	3.6701	1.6398	2.2048

Baseline Models (Reference)

Table - File Size + TPS + Avg Precision Loss

model_name	file_size_gb	bench_tps	avg_prec_loss
BF16	7.50	249.86	0.0000%
Q8_0	3.99	360.78	0.1028%
Q6_K	3.08	404.72	0.1740%
Q5_K	2.69	375.72	0.5973%
Q4_K_M	2.33	366.54	1.6668%
IQ4_NL	2.23	450.75	2.4657%
MXFP4_MOE	2.00	466.66	7.9498%

Table - PPL Columns

model_name	gen	gen_er	code	code_er	math	math_er
BF16	10.0106	0.2451	1.5917	0.0127	6.8896	0.1410
Q8_0	10.0174	0.2454	1.5931	0.0128	6.9001	0.1413
Q6_K	9.9687	0.2431	1.5927	0.0127	6.8924	0.1409
Q5_K	10.0993	0.2473	1.5978	0.0128	6.9256	0.1413
Q4_K_M	10.3239	0.2536	1.6093	0.0129	6.9423	0.1412
IQ4_NL	10.3718	0.2548	1.6125	0.0129	7.0606	0.1452
MXFP4_MOE	10.9465	0.2659	1.6645	0.0138	7.5735	0.1563

Table - Precision Loss Columns

model_name	loss_general	loss_code	loss_math
BF16	0.0000	0.0000	0.0000
Q8_0	0.0679	0.0880	0.1524
Q6_K	0.4186	0.0628	0.0406
Q5_K	0.8861	0.3832	0.5225
Q4_K_M	3.1297	1.1057	0.7649
IQ4_NL	3.6082	1.3068	2.4820
MXFP4_MOE	9.3491	4.5737	9.9266

Support

I’m a solo developer working full time for myself to achieve my dream, pouring nights and weekends into open protocols and tools that I hope make the world a little better. If you chip in, you're helping me keep the lights on while I keep shipping.

Click here to see ways to support - BTC, Paypal, GitHub sponsors.

Or, just drop a like on the repo :)