Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -19,15 +19,30 @@ NVFP4 quantization of [MiniMaxAI/MiniMax-M3](https://huggingface.co/MiniMaxAI/Mi
|
|
| 19 |
|
| 20 |
| Component | Precision |
|
| 21 |
|---|---|
|
| 22 |
-
| Routed experts
|
|
|
|
|
|
|
| 23 |
| Attention (q/k/v/o, all 60 layers) | BF16 |
|
| 24 |
| Router (`mlp.gate` + `e_score_correction_bias`) | BF16 / FP32 |
|
| 25 |
-
| Shared experts + dense-MLP `gate_up_proj` | BF16 (deliberate) |
|
| 26 |
| Embeddings, `lm_head`, all norms | BF16 |
|
| 27 |
| **Vision tower + multimodal projector (full VL stack)** | **BF16 — fully preserved** |
|
| 28 |
|
| 29 |
The repo is a complete VL model: image/video processors, chat template, and tokenizer ship alongside the weights.
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
## Reproduction
|
| 32 |
|
| 33 |
Full conversion recipe (pipeline design, split-executor amax merge, and all gotchas): **[brandonmmusic-max/minimax-m3-nvfp4-recipe](https://github.com/brandonmmusic-max/minimax-m3-nvfp4-recipe)**. A ready-to-run SM120 serving Docker image (b12x stack, MSA sparse attention, swigluoai-correct MoE) will be linked here once validated.
|
|
|
|
| 19 |
|
| 20 |
| Component | Precision |
|
| 21 |
|---|---|
|
| 22 |
+
| Routed experts — `gate_proj` / `up_proj` / `down_proj` (128 experts x 57 layers) | **NVFP4** (block-16 E2M1, FP8-E4M3 block scales, FP32 global scale) |
|
| 23 |
+
| Dense-MLP (layers 0-2) & shared-expert `down_proj` | **NVFP4** |
|
| 24 |
+
| Dense-MLP (layers 0-2) & shared-expert `gate_up_proj` | BF16 — see note below |
|
| 25 |
| Attention (q/k/v/o, all 60 layers) | BF16 |
|
| 26 |
| Router (`mlp.gate` + `e_score_correction_bias`) | BF16 / FP32 |
|
|
|
|
| 27 |
| Embeddings, `lm_head`, all norms | BF16 |
|
| 28 |
| **Vision tower + multimodal projector (full VL stack)** | **BF16 — fully preserved** |
|
| 29 |
|
| 30 |
The repo is a complete VL model: image/video processors, chat template, and tokenizer ship alongside the weights.
|
| 31 |
|
| 32 |
+
### Why dense/shared `gate_up_proj` is BF16 (an honest note, not hand-waving)
|
| 33 |
+
|
| 34 |
+
This is a scar from a calibration-config bug, kept and explained rather than papered over.
|
| 35 |
+
|
| 36 |
+
The first export used an over-broad exclusion glob `*gate*` — meant only for the MoE router (`mlp.gate`) — which silently disabled the quantizers for **every** gate-named module: the routed experts' `gate_proj` *and* the dense/shared `gate_up_proj`. That produced a 456 GB "NVFP4" export still carrying ~275 GB of un-quantized BF16 gates.
|
| 37 |
+
|
| 38 |
+
Rather than spend ~7 h recalibrating, the amaxes were recovered post-hoc — and the recovery is only exact for some of them:
|
| 39 |
+
|
| 40 |
+
- **Routed `gate_proj`** has an exact twin: its sibling `up_proj` consumes the *identical* input tensor, so the input amax is copyable bit-for-bit and the weight amax is recomputable from the source weights. -> fully **NVFP4**.
|
| 41 |
+
- **Dense/shared `gate_up_proj`** is a single *fused* `[gate; up]` linear with **no twin** to copy an input amax from. Fabricating one would be guessing, so it was left **BF16**. The `down_proj` in those same MLPs was never matched by the buggy glob, stayed calibrated, and is NVFP4 — hence the split precision within those two MLPs.
|
| 42 |
+
|
| 43 |
+
Net: the routed experts (the overwhelming majority of parameters) are fully NVFP4; only the handful of dense/shared `gate_up` projections are BF16 — conservative for quality, a few GB of size. For uniform precision, recalibrate from scratch with `calibration/data/` (the exclusion glob is fixed in the recipe).
|
| 44 |
+
|
| 45 |
+
|
| 46 |
## Reproduction
|
| 47 |
|
| 48 |
Full conversion recipe (pipeline design, split-executor amax merge, and all gotchas): **[brandonmmusic-max/minimax-m3-nvfp4-recipe](https://github.com/brandonmmusic-max/minimax-m3-nvfp4-recipe)**. A ready-to-run SM120 serving Docker image (b12x stack, MSA sparse attention, swigluoai-correct MoE) will be linked here once validated.
|