Notes from an NVFP4 port of Kimi K3
We integrated Kimi K3 into our own C++/CUDA inference engine (no PyTorch at
runtime). The released checkpoint is 1.5 TB: the routed experts are already
4-bit (MXFP4, E2M1 with one E8M0 exponent per 32 columns) and everything else is
BF16. Because MXFP4 and NVFP4 share the E2M1 grid, the experts repack
losslessly — nibbles verbatim, each E8M0 byte duplicated into the two
16-wide halves of its 32-block, the exponent bias folded into one per-tensor
power-of-two global scale. 82 432 expert files, ~20 minutes of pure I/O, no GPU
in the expert path.
Architecture facts that matter. 93 layers: layer 0 is a dense SiTU MLP,
1..92 are MoE, and the attention alternates three KDA (linear) layers followed
by one gated MLA layer, with an extra MLA layer at the end (0-based
{3,7,…,91,92}). Depth is mixed by Attention Residuals every 12 layers: each
layer's input and MLP input are a softmax mixture over the block's residual
streams plus the running prefix, scored by an RMS-dot against a learned
per-channel vector. The MoE is a latent MoE: the hidden state is projected to
3584 dims, 16 of 896 experts run there, the result is RMS-normalised and
projected back, and two shared experts see the full 7168-wide hidden. The
activation is SiTU-GLU: 4·tanh(g/4)·sigmoid(g) · 25·tanh(u/25).
Details that silently change the answer. (1) The MLA layers carryqk_rope=64 dims but no rotary module at all — mla_use_nope means those
64 dims are plain key dims, and k_rot is one head broadcast across all 96.
(2) KDA's decay is channel-wise: the checkpoint stores A_log with 128 entries
(= head_dim), and the log-decay is -5·sigmoid(exp(A_log[d])·(g_d + dt_bias))
— the fla reference indexes A_log per head, which cannot consume a
128-entry tensor for 96 heads. (3) The router selects by sigmoid+bias but
weights by the raw sigmoid renormalised to 1. (4) routed_expert_norm sits
between the expert sum and the up-projection, not before the experts. (5) The
chat format is XML, not ChatML: <|open|>message role="user"<|sep|>body… <|close|>message<|sep|><|end_of_msg|>, and the model answers inside a think
channel, so a parity harness must cut by id or window past the reasoning.
Measured. SelfCheck (framed control prompt) PASS x3 cycles, bit-exact
(The answer is Paris., logits_hash=5c1e227265d6a27b on every cycle, top-1 id 1008). Tokenizer parity 23/23
against the checkpoint's own tiktoken.model (tokenizer_kind 2 unchanged).
Tool codec verified byte-exactly against encoding_k3.py's frames on a synthetic
tokenizer. 24.5 GB pinned + 0.43 GB KDA state + 0.11 GB latent KV at 4 k
context fits the card; the latent/shared plane streams at ~103 MiB per layer
per token and the routed experts at ~27 GB per token.
Not measured (and why). The source cannot execute on one 32 GB card (it
needs fla-core plus a multi-GPU FP8/BF16 runner), so FSI/CODE/DET goldens are
template-derived (tools/kimi_k3_template_goldens.py, the prompt set's own
Expected Template) and the three gates had not been run to completion at
publication time — at the measured ~2 s/token and 512–2048-token budgets they
are multi-hour runs. Perf rows are likewise unseeded; the decode path is
host-store bound by construction (27 GB of experts per token).
correct it