gbuzhf/KAT-Coder-V2.5-Dev-APEX-MTP-GGUF

KAT-Coder-V2.5-Dev — APEX-MTP GGUF

APEX MoE-aware mixed-precision quantizations of Kwaipilot/KAT-Coder-V2.5-Dev, with the Qwen3.6-35B-A3B multi-token-prediction (MTP) head grafted in for single-file speculative decoding. Stock llama.cpp runs these files; the MTP path additionally works with builds supporting --spec-type draft-mtp.

Files

File	Size	Recipe	Fits
Kwaipilot_KAT-Coder-V2.5-Dev-MTP-UD-Q5_K_XL.gguf	27.4 GB	Highest fidelity here — see measurements	32 GB
Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Balanced.gguf	26.2 GB	mudler's KAT-native APEX Balanced config, Q8_0 MTP	32 GB
Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Compact.gguf	17.4 GB	mudler's KAT-native APEX Compact config, Q8_0 MTP	24 GB (16 GB w/ CPU-MoE)
Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Mini.gguf	14.4 GB	mudler's KAT-native APEX Mini config, Q8_0 MTP — mudler's own I-Mini has no MTP head, this one does	16 GB (12 GB w/ CPU-MoE)
Kwaipilot_KAT-Coder-V2.5-Dev-imatrix.gguf	184 MB	importance matrix used for the I-variants	—

These are text-only GGUFs. (KAT-Coder-V2.5-Dev's config declares a vision tower, but image input does not work through llama.cpp in practice, so no mmproj is shipped.)

Verify your download

sha256sum -c sha256sums.txt --ignore-missing

File	sha256
…-MTP-UD-Q5_K_XL.gguf	d178621ee21573b5b55cf972951d8f06e5e085d970a86768aa74b3a9cb838a79
…-APEX-MTP-I-Balanced.gguf	cb82e964c72108c03a1436681ab1435e5794fc80e5c6243e0afaff61c4cdd607
…-APEX-MTP-I-Compact.gguf	5c0056cbc1e4603f4abaf620678211352e37d8389e9c91f30fe8465a1957cf17
…-APEX-MTP-I-Mini.gguf	8956838bc6f481294b68ed08cff33867336552e3f36317bd7b8188f92b20ed85
…-imatrix.gguf	f5727113280b24cbacc2f5308c970ab5efc8adb15d3ac4feb0e27ae1ac469c7c

(Build provenance and toolchain versions in MANIFEST.txt.)

What's special here

1. APEX quantization (mudler & Palethorpe 2026): instead of uniform bit-width, tensors are classified by role — routed experts get a layer-wise precision gradient (sensitive edge layers high, redundant middle layers low), always-active shared experts stay at high precision (heavy-tailed distributions), attention stays at Q6_K/Q4_K/Q3_K depending on tier. On the same architecture APEX Balanced matches Q8_0 perplexity at ~⅔ the size; see the APEX technical report for full benchmarks.

2. Grafted MTP head. KAT-Coder-V2.5-Dev ships without an MTP head (mtp_num_hidden_layers: 0), but it is an architecture-identical fine-tune of Qwen3.6-35B-A3B, whose 19 mtp.* tensors were grafted into the checkpoint before conversion. llama.cpp (≥ PR #22673) bundles them as blk.40.* with nextn_predict_layers = 1; the whole draft layer is pinned at Q8_0 in every file here (an under-quantized draft head kills speculative-decode acceptance).

⚠️ Caveat: the draft head was trained against the base trunk, not KAT's fine-tuned trunk, so speculative acceptance may be lower than on stock Qwen3.6. If MTP doesn't pay off for your workload, run the same file with speculation disabled — nothing else changes.

Measured quality

KL divergence against this model's own bf16 weights — not a proxy, not an estimate. Measured on wikitext-2-raw, context 2048, 40,960 tokens. KAT's bf16 perplexity on this eval is 5.8024.

File	Size	Same top token as bf16	Max KLD	99.9% KLD	99.9% Δp
MTP-UD-Q5_K_XL	27.4 GB	96.237 ± 0.133 %	1.423	0.352	21.59 %

I-Balanced, I-Compact, and I-Mini were rebuilt on mudler's own KAT-native APEX configs (kat_coder_v25_{balanced,compact,mini}.txt) rather than the Qwen3.6 stand-in used previously (mudler had not yet published KAT-specific configs at the time), plus the MTP graft. They have not been KLD-measured against bf16 yet — treat the file sizes and recipe as current, but not the quality figures above, which apply only to MTP-UD-Q5_K_XL.

MTP-UD-Q5_K_XL

Built on Unsloth's UD-Q5_K_XL tensor allocation — all attention and SSM output at Q8_0, shared experts Q8_0, output head and embeddings Q8_0, routed experts Q5_K with the down-projection at Q6_K. That role-level design is Unsloth's work, read from their published Qwen3.6-35B-A3B GGUF and replayed here (753 tensors, transfers 1:1).

Nine tensors deviate:

• Six trunk tensors — per-layer expert protection re-derived from KAT's own importance matrix. Unsloth protect blk.34 and blk.39, which reflect the base model's activation outliers. KAT's are elsewhere: its blk.0 gate concentrates 46% of activation energy in 1% of channels, the sharpest outlier in the model. Protection was moved to blk.0, 38, 39 — a net-zero change in bytes.
• Three MTP tensors — blk.40 experts pinned Q8_0 (+266 MB). Unsloth leave theirs lower, but their draft head is native to their trunk; this one is grafted from the base onto a fine-tuned trunk, and draft acceptance is what the MTP head is for.

Uses KAT's own imatrix (computed on this model), not the base model's.

Usage

Plain llama.cpp (MTP tensors are simply carried, no speculation):

llama-server -m Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Compact.gguf \
  -ngl 99 --n-cpu-moe 30 -c 32768 -fa on

With MTP speculative decoding (builds supporting --spec-type draft-mtp, e.g. TurboQuant+):

llama-server -m Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Compact.gguf \
  --spec-type draft-mtp --spec-draft-n-max 1 \
  -ngl 99 --n-cpu-moe 32 -c 32768 -fa on

Tip from measurement: MTP alone can be a net loss on novel prose; combined with ngram-style speculation it is strongly additive on copy-heavy agent workloads (diffs, file echoes, JSON). Keep the draft chain depth at 1 when combining engines.

Build provenance

• Converted from the original bf16 safetensors (convert_hf_to_gguf.py --outtype bf16). MTP-UD-Q5_K_XL built with llama.cpp c0bc859 / apex-quant a445a12. I-Balanced / I-Compact / I-Mini built with llama.cpp 64d528b / apex-quant 4e95e47.
• I-Balanced / I-Compact / I-Mini quantized with llama-quantize --tensor-type-file using mudler's own KAT-native apex-quant configs — kat_coder_v25_balanced.txt (base Q6_K), kat_coder_v25_compact.txt (base Q4_K_M), kat_coder_v25_mini.txt (base Q3_K_M) — with the blk.40 MTP block appended (mudler's published configs don't include an MTP head).
• Importance matrix from bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF
• MTP tensors byte-copied from Qwen/Qwen3.6-35B-A3B (Apache-2.0)
• MTP bundling verified post-conversion (blk.40.nextn.eh_proj present, nextn_predict_layers=1)

Credits

• Kwaipilot for KAT-Coder-V2.5-Dev (Apache-2.0)
• Qwen team for the Qwen3.6-35B-A3B base and its MTP head (Apache-2.0)
• Ettore Di Giacinto (mudler) & Richard Palethorpe / LocalAI team for the APEX method (MIT)
• bartowski for the importance matrix
• Georgi Gerganov & contributors for llama.cpp