brandonmusic commited on
Commit
5e94c6f
·
verified ·
1 Parent(s): b517045

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +17 -2
README.md CHANGED
@@ -19,15 +19,30 @@ NVFP4 quantization of [MiniMaxAI/MiniMax-M3](https://huggingface.co/MiniMaxAI/Mi
19
 
20
  | Component | Precision |
21
  |---|---|
22
- | Routed experts (gate / up / down, 128 experts x 57 layers) | **NVFP4** (block-16 E2M1, FP8-E4M3 block scales, FP32 global scale) |
 
 
23
  | Attention (q/k/v/o, all 60 layers) | BF16 |
24
  | Router (`mlp.gate` + `e_score_correction_bias`) | BF16 / FP32 |
25
- | Shared experts + dense-MLP `gate_up_proj` | BF16 (deliberate) |
26
  | Embeddings, `lm_head`, all norms | BF16 |
27
  | **Vision tower + multimodal projector (full VL stack)** | **BF16 — fully preserved** |
28
 
29
  The repo is a complete VL model: image/video processors, chat template, and tokenizer ship alongside the weights.
30
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  ## Reproduction
32
 
33
  Full conversion recipe (pipeline design, split-executor amax merge, and all gotchas): **[brandonmmusic-max/minimax-m3-nvfp4-recipe](https://github.com/brandonmmusic-max/minimax-m3-nvfp4-recipe)**. A ready-to-run SM120 serving Docker image (b12x stack, MSA sparse attention, swigluoai-correct MoE) will be linked here once validated.
 
19
 
20
  | Component | Precision |
21
  |---|---|
22
+ | Routed experts — `gate_proj` / `up_proj` / `down_proj` (128 experts x 57 layers) | **NVFP4** (block-16 E2M1, FP8-E4M3 block scales, FP32 global scale) |
23
+ | Dense-MLP (layers 0-2) & shared-expert `down_proj` | **NVFP4** |
24
+ | Dense-MLP (layers 0-2) & shared-expert `gate_up_proj` | BF16 — see note below |
25
  | Attention (q/k/v/o, all 60 layers) | BF16 |
26
  | Router (`mlp.gate` + `e_score_correction_bias`) | BF16 / FP32 |
 
27
  | Embeddings, `lm_head`, all norms | BF16 |
28
  | **Vision tower + multimodal projector (full VL stack)** | **BF16 — fully preserved** |
29
 
30
  The repo is a complete VL model: image/video processors, chat template, and tokenizer ship alongside the weights.
31
 
32
+ ### Why dense/shared `gate_up_proj` is BF16 (an honest note, not hand-waving)
33
+
34
+ This is a scar from a calibration-config bug, kept and explained rather than papered over.
35
+
36
+ The first export used an over-broad exclusion glob `*gate*` — meant only for the MoE router (`mlp.gate`) — which silently disabled the quantizers for **every** gate-named module: the routed experts' `gate_proj` *and* the dense/shared `gate_up_proj`. That produced a 456 GB "NVFP4" export still carrying ~275 GB of un-quantized BF16 gates.
37
+
38
+ Rather than spend ~7 h recalibrating, the amaxes were recovered post-hoc — and the recovery is only exact for some of them:
39
+
40
+ - **Routed `gate_proj`** has an exact twin: its sibling `up_proj` consumes the *identical* input tensor, so the input amax is copyable bit-for-bit and the weight amax is recomputable from the source weights. -> fully **NVFP4**.
41
+ - **Dense/shared `gate_up_proj`** is a single *fused* `[gate; up]` linear with **no twin** to copy an input amax from. Fabricating one would be guessing, so it was left **BF16**. The `down_proj` in those same MLPs was never matched by the buggy glob, stayed calibrated, and is NVFP4 — hence the split precision within those two MLPs.
42
+
43
+ Net: the routed experts (the overwhelming majority of parameters) are fully NVFP4; only the handful of dense/shared `gate_up` projections are BF16 — conservative for quality, a few GB of size. For uniform precision, recalibrate from scratch with `calibration/data/` (the exclusion glob is fixed in the recipe).
44
+
45
+
46
  ## Reproduction
47
 
48
  Full conversion recipe (pipeline design, split-executor amax merge, and all gotchas): **[brandonmmusic-max/minimax-m3-nvfp4-recipe](https://github.com/brandonmmusic-max/minimax-m3-nvfp4-recipe)**. A ready-to-run SM120 serving Docker image (b12x stack, MSA sparse attention, swigluoai-correct MoE) will be linked here once validated.