ScarletKc-2.0 NVFP4 (W4A16)

W4A16 NVFP4 checkpoint derived from BF16 ScarletKc-2.0 with ModelOpt Local Hessian post-training quantization (PTQ) followed by scale-only quantization-aware distillation (QAD). The upstream model card describes the base model, its lineage, licence and usage. ScarletKc/ScarletKc-2.0 at ddebd3b2

Property Value
Base model ScarletKc/ScarletKc-2.0 @ ddebd3b2d5f6a91e1d9491c3dc2982551ffb3340 (bf16 source index: 1,199 tensors = 850 language + 333 vision + 15 MTP + 1 other (lm_head))
Quantization W4A16 NVFP4 — weights NVFP4 (4-bit, group size 16), activations are not quantized (bf16 in the quantization path, fp16 at serving time)
Method Layerwise ModelOpt Local Hessian PTQ with E4M3 block-scale search, followed by scale-only QAD
Quantized modules 256 language linears (every mlp.gate_proj / mlp.up_proj / mlp.down_proj, plus q/k/v/o_proj of the 16 full-attention layers)
Kept bf16 (verbatim, per-tensor SHA-256 match) embeddings, lm_head, all layer norms, the 48 linear-attention (Mamba-style) blocks, the whole vision tower (333 tensors) and the MTP head (15 tensors)
Weights layout 13 safetensors shards + model-mtp.safetensors; published index: 1,711 tensors = 1,362 language + 333 vision + 15 MTP + 1 other, of which 256 are NVFP4 weight_scale tensors (one per quantized linear); 28,551,539,168 bytes of tensors
Context 262,144 native tokens; 409,600 with the published YaRN override (runtime/config-yarn-409600.json)
Calibration 512 samples × 2,048 tokens = 1,048,576 real tokens per quantized linear, batch size 1, no padding
Quantization host 1 × Tesla V100-SXM2-32GB (SM70), shared allocation; bf16 math emulated in fp32 with bf16-rounded outputs
Serving runtime 1Cat-vLLM 1.5.4, two Tesla V100s, TP2, FP16 compute, E4M3 KV cache, MTP4

The adaptation initialized its quantizers from the Local Hessian PTQ checkpoint with artifact identity 16009f620fbf798b32d310b4ddf88ea38a4aeb79145441b55ae882c2a2c54bf2. The BF16 base checkpoint supplied the teacher distributions. The adaptation changed selected E4M3 block scales while freezing source weights and global multipliers. Validation selected step 30. On 15 independent held-out answers with 708 assistant tokens, KL to BF16 decreased from 0.010747 to 0.010185 nats/token and NLL decreased from 0.301590 to 0.301274 versus PTQ. The export changed 14 packed tensors across seven projections and preserved the other 1,697 tensors byte-for-byte. Adaptation method

The manifest lists SHA-256 digests for published model and documentation files. MANIFEST.sha256

PTQ initialization

The producer receipt — not the plan — is the authority. The executed run is recorded in a receipt whose content hash is 537e6e29dd5b788473e597bfc576ebdb4e09f2dd26f40e01b6118d4f9ccbc1d1; its sanitized projection is published as quantization/quantization-method.json.

  • Algorithm: method = local_hessian, block_size = 16, hessian_input_dtype = float32, layerwise calibration enabled with calib_mutates_weights = false and QDQ activations taken from the previous layer, fp8_scale_sweep = true over 126 E4M3 scale candidates per block.
  • Format: W4A16_NVFP4, group_size = 16, activation_quantization = false, weight-scale dtype float8_e4m3fn, scale convention multiplier. Activations are not quantized: the quantization path carried bf16 activations (emulated in fp32 on the SM70 host), while the serving profile runs fp16 compute (--dtype half). Those are two different numeric paths; the export was produced under the first, and the repository does not claim bit-level equivalence between them.
  • Export path: NVIDIA Model-Optimizer @ ee4782b717450f04c3dcfe7616cbc4c1ed8e38ce (unified_export_hf.py, quant_utils.py, convert_hf_config.py pinned by SHA-256). The pinned Triton 3.6 FP4 kernels cannot compile for SM70, so the NVFP4 encoder, the Hessian estimator and the scalar-exact CPU emulation were transcribed locally from that pinned ModelOpt tree (quantization/recipe/nvfp4_torch.py, quantization/recipe/bf16_math.py).
  • Sweep backend: the portable torch Hessian backend was selected with row_chunk = 4096, candidate_chunk = 126. The native SM70 backend failed its numeric gate on 94 of 10,240 blocks in the first representative shape; the portable backend matched the independent FP32/F64 oracles exactly on all 79,875 tested blocks and on every tie/zero fixture.
  • Numerics on this host: bf16 was emulated as fp32 compute with bf16-rounded outputs (bf16_math.compute_dtype = float32, attention backend SDPBackend.MATH). This is not the same numerics as a native bf16 GPU run; the exported bytes are the bytes this procedure produced.
  • PTQ froze all source weights while estimating Local Hessians and selecting block scales. The upstream ScarletKc-2.0 BF16 model used an affective orthogonal projection and on-policy identity LoRA before this quantization.

Full disclosure of the executed algorithm, calibration and environment: quantization/quantization-method.json.

Calibration

  • 512 packed sequences × 2,048 tokens = 1,048,576 real tokens per quantized linear, batch size 1, no padding tokens.

  • The per-dataset quotas below are packed 2,048-token window quotas, not source-row counts: the materializer fills each split's quota with whole windows and stops when the quota is met.

  • 334 source records were accepted in total: the first accepted record of each of the 32 splits was reserved for holdout (32 holdout conversations, heldout.jsonl, heldout-requests.jsonl, never used for scale search), and the remaining 302 records became the calibration conversations (selected-records.jsonl), which were packed into exactly 512 windows of 2,048 tokens (1,048,576 real tokens, zero padding).

  • Conversation-level dedup and template-error rejection are implemented as guards. No rejection was actually recorded in this materialization — every per-split rejection map is empty and the selection-events log is empty — so the observed shrinkage was zero. Windows, conversations, records and source rows are four different counts and are not interchangeable.

  • Composition: the official nemotron-post-training-v3 source set, pinned per dataset revision:

    Dataset Revision Packed 2,048-token windows
    nvidia/Nemotron-SFT-Instruction-Following-Chat-v2 1a9454ed054b8544503ab8d8c0a519d141a44c5b 74
    nvidia/Nemotron-Science-v1 82e1af468197076b4f0f392c239274eac032adc7 73
    nvidia/Nemotron-Competitive-Programming-v1 d6e7c6b404ed5db6e1104b41d0f80a0c7dad7bf8 73
    nvidia/Nemotron-SFT-Agentic-v2 7c804833427f633ccd53b582dbf02525fd680f78 73
    nvidia/Nemotron-Math-v2 8e793210e175b6406c752a870f585f62de98c0d3 73
    nvidia/Nemotron-SFT-SWE-v2 bd151f3f2d89c4804dda0083d912bd9f6a0a9fb7 73
    nvidia/Nemotron-SFT-Multilingual-v1 22c86505762a7c595abee309d720084351c9f4ba 73
  • Selection is deterministic: immutable raw JSONL order, per-dataset window quotas, split-local packing, the first accepted record of each split reserved for holdout, conversation-level dedup, no padding. This adapts the official seven-source composition; it is not an exact reproduction of NVIDIA's own token selection.

  • Coverage limits recorded at materialization time: no images and no audio; only the agentic search split of the agentic dataset (not complete tool-use coverage).

  • The calibration set was materialized once, on CPU, before quantization and then consumed byte-identically (binding kind unchanged_materialized_calibration); no re-selection happened during the quantize run.

Reproducing the quantization

The byte-exact recipe that produced the PTQ initialization is published under quantization/recipe/ (23 files, aggregate SHA-256 914adf7146a967bd87b5e4a7888369544c725d82d3f8fabdfdefcbd1b56c3ec7, the same identity the calibration set is bound to). Pinned dependencies: Model-Optimizer ee4782b717450f04c3dcfe7616cbc4c1ed8e38ce, 1Cat-vLLM 1.5.3 cp312/cu128 wheel 0558021651c41ab111609d121cef029c09224de4b6e6f8fd362754dfdd84990f, Torch 2.10.0+cu128.

Steps: bootstrap.sh → preflight.py source → calibration.py (materialize the frozen 512-sample set) → quantize.py (PTQ + ModelOpt export) → assemble.py (shard the export) → preflight.py export.

Note on the recipe copies: this repository publishes the executed revision. A later, path-redacted variant of the same recipe (aggregate 282ec96f7f8a7fbbe7efb454e7dc93026bc45aa19dfb5d0daae4255b8fe1e5e3) differs in four files — commands.json, resources.json, run_quantize.sh (private paths replaced by placeholders, later state string, assembly step relocated) and nvfp4_torch.py (one provenance constant that names the ModelOpt source tree: local path vs. canonical GitHub URL). Every other file is byte-identical, including all quantization-algorithm sources; nvfp4_torch.py differs only in that non-algorithmic locator constant, so the two revisions implement the same algorithm.

Serving on V100 with 1Cat-vLLM 1.5.4

The validated configuration uses two Tesla V100 32 GB GPUs with TP2, FP16 compute, E4M3 KV cache and MTP4. It enables regular prefix caching and disables partial prefix caching and SM70 sparse target sampling. 1Cat-vLLM 1.5.4 release

mkdir -p /opt/scarlet-yarn && cp runtime/config-yarn-409600.json /opt/scarlet-yarn/config.json
bash runtime/serve-1cat-vllm-v100.sh "$PWD" /opt/scarlet-yarn

The launcher selects the validated serving template and binds sampling defaults to the model directory. Runtime flags and requirements

Why --generation-config is mandatory here

The first serving attempt pointed --hf-config-path at the YaRN override directory, which contains only config.json. vLLM then auto-discovered no generation_config.json from that directory and fell back to library defaults (temperature 1.0 / top_p 1.0 / top_k 0) instead of the model's temperature 0.6 / top_p 0.9 / top_k 20. With the model directory bound explicitly via --generation-config <model directory>, the model's own generation config is used and the sampling matrix behaves as intended. The weight bytes were identical in both runs, which isolates the generation-config binding as the cause of the behavioural difference. That is a statement about this comparison only; it is not a general claim about the quality of the quantized weights.

PTQ baseline serving results

Evidence is a sanitized aggregate of the corrected 10-case matrix plus the paired bf16-source control; raw receipts, prompts and server logs are not published (quantization/quality-evidence.json).

Check Result
Corrected sampling matrix (5 sampling cells × 2 rounds, Anthropic /v1/messages, streaming) 10/10 completed with stop_reason = end_turn, no protocol errors, all answers and reasoning traces manually reviewed as coherent, no multilingual degeneration
Same matrix before the generation-config fix 8/10 completed. The two "server default sampling" cells (no explicit sampling parameters) streamed degenerate multilingual output (mixed-script degeneration in the reasoning and answer text) and never finished inside the 60 s request budget: the client timed out, no final message-stop event arrived, and the final usage counters were therefore unavailable (the message_start usage reported output_tokens = 0 although content had been streamed). Both outputs were inspected in review and both were rejected
Paired control, same server software, BF16 source weights served with FP16 compute (--dtype half) vs the PTQ baseline, 5 synthetic cases (EN + zh) 5/5 end_turn, all bounded-pass, corrupted_text = false in both arms. Input length was 15,873–15,921 tokens. The bounded profile used target-only decoding, FP16 KV cache, a 32,768-token window, one sequence and eager mode.
Sampling values resolved by the server after the fix temperature 0.6 / top_p 0.9 / top_k 20 (model generation_config.json)

Context window: what was actually tested

  • The configured window is 409,600 tokens (YaRN factor 1.5625 over the native 262,144-token window).
  • The adapted checkpoint passed 10/10 reviewed Anthropic streams on 1Cat-vLLM 1.5.4 at 15,630–17,109 input tokens. Direct and gateway streaming also passed after the production switch. Generation quality at the full configured window remains unmeasured. Adaptation validation

Limits and honest caveats

  • Adaptation changes block scales in a bounded projection subset. A full-weight QAT run and a per-layer or MSE-vs-Hessian ablation remain unmeasured.
  • Calibration budget. 512 × 2,048 tokens is a modest calibration set chosen before execution and never revisited; larger or per-layer-adaptive calibration was not evaluated.
  • Host numerics. The export was produced on SM70 with bf16 emulated in fp32. A native bf16 GPU export could differ in the last bits.
  • Quality scope. The evidence above is a small synthetic suite (10 matrix cases, 5 paired control cases) with manual review — not a benchmark, and not a regression suite.
  • This repository is the W4A16 artifact. It is a different checkpoint from any W4A4/FP8-mixed NVFP4 build of the same base model; do not mix the two when comparing sizes or quality.

Files

Path Content
model-0000{1..13}-of-00013.safetensors, model-mtp.safetensors Adapted NVFP4 export and preserved MTP weights
model.safetensors.index.json shard index (1,711 tensors, incl. the 15 MTP tensors)
config.json, hf_quant_config.json, generation_config.json model config, quantization config (256 quantized layers), sampling defaults
tokenizer.json, tokenizer_config.json, chat_template.jinja, processor_config.json tokenizer, chat template, multimodal processor
LICENSE, NOTICE upstream Apache-2.0 licence and notice, byte-identical to the source repository
MANIFEST.sha256 SHA-256 of every published file
quantization/recipe/ byte-exact executed recipe (23 files)
quantization/quantization-method.json executed algorithm, environment, calibration binding (sanitized)
quantization/calibration-provenance.json calibration datasets, revisions, quotas, holdout, hashes (sanitized)
quantization/execution-results.json aggregate execution/verification history (sanitized)
quantization/quality-evidence.json PTQ baseline results from the corrected sampling matrix and the paired control
quantization/adaptation-method.md Scale adaptation design, implementation checks and candidate results
runtime/ YaRN override, serving template and portable 1.5.4 launcher

Licence

Apache License 2.0, inherited from ScarletKc/ScarletKc-2.0 (and from JonathanColetti/Qwen3.8-27B-Uncensored / Qwen/Qwen3.8-27B). See LICENSE and NOTICE. The reduced refusal behaviour described in the upstream card applies to this checkpoint as well; add external moderation before exposing it to untrusted users.

Downloads last month
101
Safetensors
Model size
28B params
Tensor type
BF16
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for cyjin-yl/ScarletKc-2.0-NVFP4

Base model

Qwen/Qwen3.8-27B
Quantized
(7)
this model