Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588

RK3588 NPU: Prepared for open_rknpu2_lm_infer, an open-source inference engine for RK3588. This is a compressed-tensors safetensors checkpoint, not an RKLLM-format model.

RK3588 NPU๏ผš ๆœฌๆจกๅž‹้ขๅ‘ๅผ€ๆบๆŽจ็†ๅผ•ๆ“Ž open_rknpu2_lm_inferใ€‚ๆจกๅž‹้‡‡็”จ compressed-tensors safetensors ๆ ผๅผ๏ผŒไธๆ˜ฏ RKLLM ๆ ผๅผใ€‚

Quick start / ๅฟซ้€Ÿๅผ€ๅง‹

On an RK3588 Linux device, install the engine using the project instructions, then download and run:

ๅœจ RK3588 Linux ่ฎพๅค‡ไธŠๆŒ‰้กน็›ฎ่ฏดๆ˜Žๅฎ‰่ฃ…ๅผ•ๆ“Ž๏ผŒ็„ถๅŽไธ‹่ฝฝๅนถ่ฟ่กŒ๏ผš

hf download happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 --local-dir models/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588
./bin/open_rknpu2_lm_infer --model models/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 chat

Other inference engines / ๅ…ถไป–ๆŽจ็†ๅผ•ๆ“Ž

vLLM and SGLang support compressed-tensors in some configurations. You can try this checkpoint with compatible releases and hardware using vllm serve happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 or python -m sglang.launch_server --model-path happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588. Support for this exact mixed INT4/INT8 configuration, model architecture, and hardware varies by engine version and backend; execution and accuracy have not been verified here. See the model-specific limitations below.

vLLM ๅ’Œ SGLang ๅœจ้ƒจๅˆ†้…็ฝฎไธ‹ๆ”ฏๆŒ compressed-tensors๏ผŒๅฏๅœจๅ…ผๅฎน็š„็‰ˆๆœฌๅ’Œ็กฌไปถไธŠๅฐ่ฏ•ไธŠ่ฟฐๅ‘ฝไปคใ€‚ๅ…ทไฝ“่ƒฝๅฆ่ฟ่กŒๅ–ๅ†ณไบŽๅผ•ๆ“Ž็‰ˆๆœฌใ€ๅŽ็ซฏใ€ๆจกๅž‹ๆžถๆž„ๅŠๆญคๆททๅˆ INT4/INT8 ้…็ฝฎ๏ผ›ๆœฌๆจกๅž‹ๅœจ่ฟ™ไธคไธชๅผ•ๆ“ŽไธŠ็š„่ฟ่กŒๅ’Œ็ฒพๅบฆๅฐšๆœชๅœจๆญค้ชŒ่ฏใ€‚ๆจกๅž‹็‰นๅฎš้™ๅˆถ่งไธ‹ๆ–‡ใ€‚

English ยท ไธญๆ–‡

English

Model

A mixed W4A8/W8A8 quantization of Qwen/Qwen3.5-4B, using the C7 fixed offline residual rotation + QQQ recipe. Weights are symmetric per-output-channel INT4/INT8, with FP32 scales and dynamic per-token INT8 activations. The safetensors payload is 4.645 GB (4.326 GiB), including the BF16 vision and MTP branches.

Property Value
Architecture Qwen3_5ForConditionalGeneration, 32 text layers
Source ModelScope Qwen/Qwen3.5-4B
Source weights revision fcb1a040bb418b0b8add6f6f6c475386abc2cb97
Quantized Linear layers 197 W4A8 + 52 W8A8
Decoder Linear weight elements in W4 97.91%
Packing compressed-tensors pack-quantized, safetensors
Residual rotation Signed Hadamard 20ร—128, seed 42, folded offline
QQQ Per-channel, block size 128, damping 0.01
Remaining parameters BF16

The W8A8 projections are lm_head, in_proj_a and in_proj_b in all 24 linear-attention layers, and MLP down_proj in text layers 1, 14, 28 (zero-based). All other quantized text Linear layers are W4A8. The exact names are in bitwidth_manifest.json. The main decoder requires no online rotation. Qwen3.5's 1 + weight RMSNorm gains are folded into projections. The vision merger output is rotated to match the text residual basis.

Accuracy

Model WikiText PPL โ†“ CMMLU /201 โ†‘ GSM8K /100 โ†‘ GSM8K truncations โ†“
Original weights 9.944655 158 93 2
C7 W4A8/W8A8 10.476089 151 95 1

Relative PPL increase: 5.34%. These results use the project's INT8ร—INT8โ†’INT32 reference GEMM, symmetric full-K per-token maxabs/127 activation scaling, FP32 nonlinearities and KV states, and disabled TF32. The original BF16 weights are evaluated in FP32. Standard compressed-tensors loading uses its own activation scaling.

WikiText uses all 580 complete 512-token windows of the retokenized raw test text, scoring target positions 257โ€“511: 147,900 targets. CMMLU uses 3 fixed test questions from each of 67 subjects, seed 42, zero-shot native chat, and next-token A/B/C/D logits. GSM8K uses 100 fixed test examples, seed 42, greedy decoding up to 512 new tokens, and numeric exact match; truncated answers count as incorrect. Both chat evaluations use enable_thinking=False. See evaluation.json and evaluation/ for protocols and individual results.

Calibration uses 256 complete no-thinking conversations: 128 Chinese, 64 English, and 64 math, totaling 97,794 tokens without truncation. Preprocessed weights are BF16; QQQ forwards and Hessians are FP32, with activation quantization disabled during calibration. See calibration.json and selection_recipe.json.

Transformers loading

Install the versions in requirements.txt. Standard Transformers loading, full forward execution, and cached text generation passed CPU FP32 verification.

import torch
from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration

repo = "happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    repo, dtype=torch.float32, device_map="cpu", attn_implementation="sdpa"
).eval()
ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": "่ฏท่ฎก็ฎ— 12+7๏ผŒๅช่พ“ๅ‡บ็ป“ๆžœใ€‚"}],
    tokenize=True, return_tensors="pt", return_dict=False,
    add_generation_prompt=True, enable_thinking=False,
)
with torch.inference_mode():
    output = model.generate(
        input_ids=ids, attention_mask=torch.ones_like(ids),
        max_new_tokens=32, do_sample=False, use_cache=True, logits_to_keep=1,
        eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(output[0, ids.shape[1]:], skip_special_tokens=True))

MTP

All 15 original MTP tensors are preserved bit-exact in BF16 and indexed in model-mtp.safetensors. The MTP branch shares the main embedding and output head. Its acceptance results on 32 fixed prompts, greedy one-token proposals and at most 256 new tokens, are:

Model / MTP connection Accepted / proposed Acceptance
Original weights 3249 / 3864 84.08%
C7 with coordinate alignment 3249 / 3963 81.98%
C7 without coordinate alignment 36 / 7162 0.50%

The aligned result loses 2.10 percentage points against the original. These are integer reference runtime measurements, with BF16 MTP computation and a shared W8A8 head. Prompts comprise 16 math, 8 Chinese explanation, and 8 English generation tasks. Both target and MTP use cached decoding; acceptance includes the verifier bonus schedule. Detailed records are in evaluation/mtp/.

mtp_coordinate_adapter.json records the external alignment needed between the rotated target and the original MTP. For row-vector states, rotation R, and original final RMSNorm gain gamma, the transforms are:

MTP embedding input = e_rot @ R.T
MTP hidden input    = (h_postnorm_rot @ R.T) * gamma
Shared head input   = (z_mtp / gamma) @ R

Limitations

  • This C7 checkpoint does not provide native vLLM/SGLang MTP support. The adapter JSON is project metadata, not an adapter format those engines consume. The 81.98% result requires the external transforms above; retaining BF16 MTP weights alone gives the unaligned result. Standard Transformers text generation uses the ordinary autoregressive path.
  • CUDA vLLM/SGLang execution of this packed W4A8 format is unverified. The checkpoint differs from W4A16, groupwise GPTQ/AWQ, GGUF, and QQQ's native kernel format.
  • Full benchmark accuracy under stock compressed-tensors scaling and production throughput have not been measured. CMMLU/GSM8K scores are fixed subsets.
  • Image/video accuracy, thinking mode, long context, and sampled or multi-candidate MTP behavior have not been evaluated.

License

Apache-2.0. Original model by the Qwen team, Alibaba Cloud; this is an independent quantized derivative. Source file revisions are recorded in source_provenance.json. See LICENSE and NOTICE.

ไธญๆ–‡

ๆœฌๆจกๅž‹ๆ˜ฏ Qwen3.5-4B ็š„ C7 W4A8/W8A8 ๆททๅˆ้‡ๅŒ–็‰ˆ๏ผŒ้‡‡็”จๅ›บๅฎš็ฆป็บฟๆฎ‹ๅทฎๆ—‹่ฝฌ ไธŽ QQQ๏ผš197 ไธช W4A8 Linearใ€52 ไธช W8A8 Linear๏ผŒdecoder Linear ๅ‚ๆ•ฐไธญ 97.91% ไธบ W4ใ€‚้€่พ“ๅ‡บ้€š้“ๅฏน็งฐๆƒ้‡ใ€FP32 scaleใ€้€ token ๅŠจๆ€ INT8 ๆฟ€ๆดป๏ผŒ compressed-tensors ๆ‰“ๅŒ…๏ผŒtensor ๅ…ฑ 4.645 GB๏ผŒๅŒ…ๅซ BF16 ่ง†่ง‰ๅˆ†ๆ”ฏๅ’Œ MTPใ€‚

ๅŽŸๅง‹ๆƒ้‡ไปŽ ModelScope ไธ‹่ฝฝ๏ผŒ็‰ˆๆœฌไธบ fcb1a040bb418b0b8add6f6f6c475386abc2cb97ใ€‚ W8 ่Œƒๅ›ดไธบ่พ“ๅ‡บๅคดใ€24 ๅฑ‚็บฟๆ€งๆณจๆ„ๅŠ›็š„ in_proj_a/in_proj_b๏ผŒไปฅๅŠ็ฌฌ 1ใ€14ใ€28 ๅฑ‚ MLP down_projใ€‚ๆฎ‹ๅทฎไฝฟ็”จ seed 42 ็š„ signed Hadamard 20ร—128 ๆ—‹่ฝฌ๏ผŒไธปๅนฒๆ— ๅœจ็บฟๆ—‹่ฝฌใ€‚ ๆ กๅ‡†ไฝฟ็”จ 256 ๆกๅฎŒๆ•ดๅฏน่ฏ๏ผŒไธญๆ–‡ 128ใ€่‹ฑๆ–‡ 64ใ€ๆ•ฐๅญฆ 64๏ผŒๅ…ฑ 97,794 token๏ผŒๆ— ๆˆชๆ–ญใ€‚

ๆจกๅž‹ WikiText PPL โ†“ CMMLU /201 โ†‘ GSM8K /100 โ†‘ ๆ•ฐๅญฆๆˆชๆ–ญๆ•ฐ โ†“
ๅŽŸๅง‹ๆƒ้‡ 9.944655 158 93 2
C7 W4A8/W8A8 10.476089 151 95 1

PPL ็›ธๅฏนไธŠๅ‡ **5.34%**ใ€‚็ป“ๆžœๆฅ่‡ช INT8ร—INT8โ†’INT32 ๅ‚่€ƒ GEMM๏ผŒๆฟ€ๆดปๆŒ‰ full-K ้€ token maxabs/127 ้‡ๅŒ–๏ผŒ้ž็บฟๆ€งไธŽ KV ๆŒ‰ FP32 ่ฎก็ฎ—๏ผŒๅ…ณ้—ญ TF32ใ€‚ WikiText ไธบๅ…จ้ƒจ 580 ไธช 512-token ๅ—ใ€147,900 ไธช่ฎกๅˆ† token๏ผ›CMMLU ไธบ 67 ็ง‘ๅ„ 3 ้ข˜๏ผ› GSM8K ไธบๅ›บๅฎš 100 ้ข˜๏ผŒgreedy ๆœ€ๅคš 512 ไธชๆ–ฐ token๏ผŒๆˆชๆ–ญ่ฎก้”™๏ผŒๅ…ณ้—ญ thinkingใ€‚ ้€้ข˜็ป“ๆžœใ€็ฒพ็กฎไฝๅฎฝๆธ…ๅ•ๅŠๆƒ้‡้ชŒๆ”ถ่ฎฐๅฝ•ๅทฒ้šไป“ๅบ“ๆไพ›ใ€‚

ๆ™ฎ้€š่‡ชๅ›žๅฝ’็”Ÿๆˆๅทฒ้€š่ฟ‡ๆ ‡ๅ‡† Transformers FP32 CPU ๅŠ ่ฝฝ้ชŒๆ”ถ๏ผŒไพ่ต–ๅ’Œไปฃ็ ่งไธŠๆ–‡ใ€‚ ๆ ‡ๅ‡† compressed-tensors ็š„้ป˜่ฎคๆฟ€ๆดป็ผฉๆ”พไธŽไธŠ่ฟฐๅ‚่€ƒ่ฏ„ๆต‹ๅฃๅพ„ไธๅŒใ€‚

MTP ็š„ 15 ไธชๅผ ้‡ไฟๆŒๆบๆจกๅž‹ BF16 ๅŽŸๅ€ผใ€‚32 ๆกๅ›บๅฎšๆ็คบ็š„ๅ‚่€ƒๆŽฅๅ—็އ๏ผšๅŽŸๆจกๅž‹ **84.08%**๏ผŒC7 ๅŠ ๅๆ ‡้€‚้… **81.98%**๏ผŒไธ‹้™ 2.10 ไธช็™พๅˆ†็‚น๏ผ›ไธๅŠ ้€‚้…ไธบ **0.50%**ใ€‚ ๅๆ ‡ๅ˜ๆขๅ’Œๅ‚ๆ•ฐ่งไธŠๆ–‡ๅŠ mtp_coordinate_adapter.jsonใ€‚

ๅฑ€้™๏ผšๆœฌ C7 ๆƒ้‡ๅฐšไธๆ”ฏๆŒ็›ดๆŽฅ็”จๅŽŸ็”Ÿ vLLM/SGLang ๅผ€ๅฏ MTP๏ผŒ่ฏฅ JSON ไธๆ˜ฏๅผ•ๆ“Ž ๆ”ฏๆŒ็š„้€‚้…ๆ ผๅผ๏ผ›81.98% ไธบๅŒ…ๅซๅค–้ƒจๅๆ ‡ๅ˜ๆข็š„ๆ•ดๆ•ฐๅ‚่€ƒ็ป“ๆžœใ€‚่ฏฅ packed W4A8 ๆ ผๅผ็š„ CUDA vLLM/SGLang ่ฟ่กŒใ€็”Ÿไบงๅžๅใ€ๆ ‡ๅ‡† compressed-tensors ๅฎŒๆ•ด็ฒพๅบฆใ€ๅ›พๅƒ/่ง†้ข‘ใ€ thinkingใ€้•ฟไธŠไธ‹ๆ–‡ๅ’Œ้‡‡ๆ ท MTP ๅฐšๆœช้ชŒๆ”ถใ€‚

Downloads last month
147
Safetensors
Model size
5B params
Tensor type
I32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588

Finetuned
Qwen/Qwen3.5-4B
Quantized
(680)
this model