Instructions to use happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588") model = AutoModelForMultimodalLM.from_pretrained("happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588
- SGLang
How to use happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 with Docker Model Runner:
docker model run hf.co/happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588
Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588
RK3588 NPU: Prepared for open_rknpu2_lm_infer, an open-source inference engine for RK3588. This is a compressed-tensors safetensors checkpoint, not an RKLLM-format model.
RK3588 NPU๏ผ ๆฌๆจกๅ้ขๅๅผๆบๆจ็ๅผๆ open_rknpu2_lm_inferใๆจกๅ้็จ compressed-tensors safetensors ๆ ผๅผ๏ผไธๆฏ RKLLM ๆ ผๅผใ
Quick start / ๅฟซ้ๅผๅง
On an RK3588 Linux device, install the engine using the project instructions, then download and run:
ๅจ RK3588 Linux ่ฎพๅคไธๆ้กน็ฎ่ฏดๆๅฎ่ฃ ๅผๆ๏ผ็ถๅไธ่ฝฝๅนถ่ฟ่ก๏ผ
hf download happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 --local-dir models/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588
./bin/open_rknpu2_lm_infer --model models/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 chat
Other inference engines / ๅ ถไปๆจ็ๅผๆ
vLLM and SGLang support compressed-tensors in some configurations. You can try this checkpoint with compatible releases and hardware using vllm serve happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588 or python -m sglang.launch_server --model-path happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588. Support for this exact mixed INT4/INT8 configuration, model architecture, and hardware varies by engine version and backend; execution and accuracy have not been verified here. See the model-specific limitations below.
vLLM ๅ SGLang ๅจ้จๅ้ ็ฝฎไธๆฏๆ compressed-tensors๏ผๅฏๅจๅ ผๅฎน็็ๆฌๅ็กฌไปถไธๅฐ่ฏไธ่ฟฐๅฝไปคใๅ ทไฝ่ฝๅฆ่ฟ่กๅๅณไบๅผๆ็ๆฌใๅ็ซฏใๆจกๅๆถๆๅๆญคๆททๅ INT4/INT8 ้ ็ฝฎ๏ผๆฌๆจกๅๅจ่ฟไธคไธชๅผๆไธ็่ฟ่กๅ็ฒพๅบฆๅฐๆชๅจๆญค้ช่ฏใๆจกๅ็นๅฎ้ๅถ่งไธๆใ
English
Model
A mixed W4A8/W8A8 quantization of Qwen/Qwen3.5-4B, using the C7 fixed offline residual rotation + QQQ recipe. Weights are symmetric per-output-channel INT4/INT8, with FP32 scales and dynamic per-token INT8 activations. The safetensors payload is 4.645 GB (4.326 GiB), including the BF16 vision and MTP branches.
| Property | Value |
|---|---|
| Architecture | Qwen3_5ForConditionalGeneration, 32 text layers |
| Source | ModelScope Qwen/Qwen3.5-4B |
| Source weights revision | fcb1a040bb418b0b8add6f6f6c475386abc2cb97 |
| Quantized Linear layers | 197 W4A8 + 52 W8A8 |
| Decoder Linear weight elements in W4 | 97.91% |
| Packing | compressed-tensors pack-quantized, safetensors |
| Residual rotation | Signed Hadamard 20ร128, seed 42, folded offline |
| QQQ | Per-channel, block size 128, damping 0.01 |
| Remaining parameters | BF16 |
The W8A8 projections are lm_head, in_proj_a and in_proj_b in all 24 linear-attention
layers, and MLP down_proj in text layers 1, 14, 28 (zero-based). All other quantized
text Linear layers are W4A8. The exact names are in bitwidth_manifest.json.
The main decoder requires no online rotation. Qwen3.5's 1 + weight RMSNorm gains
are folded into projections. The vision merger output is rotated to match the text residual basis.
Accuracy
| Model | WikiText PPL โ | CMMLU /201 โ | GSM8K /100 โ | GSM8K truncations โ |
|---|---|---|---|---|
| Original weights | 9.944655 | 158 | 93 | 2 |
| C7 W4A8/W8A8 | 10.476089 | 151 | 95 | 1 |
Relative PPL increase: 5.34%. These results use the project's INT8รINT8โINT32
reference GEMM, symmetric full-K per-token maxabs/127 activation scaling, FP32
nonlinearities and KV states, and disabled TF32. The original BF16 weights are
evaluated in FP32. Standard compressed-tensors loading uses its own activation scaling.
WikiText uses all 580 complete 512-token windows of the retokenized raw test text,
scoring target positions 257โ511: 147,900 targets. CMMLU uses 3 fixed test questions
from each of 67 subjects, seed 42, zero-shot native chat, and next-token A/B/C/D logits.
GSM8K uses 100 fixed test examples, seed 42, greedy decoding up to 512 new tokens,
and numeric exact match; truncated answers count as incorrect. Both chat evaluations
use enable_thinking=False. See evaluation.json and
evaluation/ for protocols and individual results.
Calibration uses 256 complete no-thinking conversations: 128 Chinese, 64 English,
and 64 math, totaling 97,794 tokens without truncation. Preprocessed weights are BF16;
QQQ forwards and Hessians are FP32, with activation quantization disabled during calibration.
See calibration.json and selection_recipe.json.
Transformers loading
Install the versions in requirements.txt. Standard Transformers
loading, full forward execution, and cached text generation passed CPU FP32 verification.
import torch
from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration
repo = "happyme531/Qwen3.5-4B-W4A8-Mixed-QQQ-Rot-RK3588"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
repo, dtype=torch.float32, device_map="cpu", attn_implementation="sdpa"
).eval()
ids = tokenizer.apply_chat_template(
[{"role": "user", "content": "่ฏท่ฎก็ฎ 12+7๏ผๅช่พๅบ็ปๆใ"}],
tokenize=True, return_tensors="pt", return_dict=False,
add_generation_prompt=True, enable_thinking=False,
)
with torch.inference_mode():
output = model.generate(
input_ids=ids, attention_mask=torch.ones_like(ids),
max_new_tokens=32, do_sample=False, use_cache=True, logits_to_keep=1,
eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0, ids.shape[1]:], skip_special_tokens=True))
MTP
All 15 original MTP tensors are preserved bit-exact in BF16 and indexed in
model-mtp.safetensors. The MTP branch shares the main
embedding and output head. Its acceptance results on 32 fixed prompts, greedy
one-token proposals and at most 256 new tokens, are:
| Model / MTP connection | Accepted / proposed | Acceptance |
|---|---|---|
| Original weights | 3249 / 3864 | 84.08% |
| C7 with coordinate alignment | 3249 / 3963 | 81.98% |
| C7 without coordinate alignment | 36 / 7162 | 0.50% |
The aligned result loses 2.10 percentage points against the original. These are
integer reference runtime measurements, with BF16 MTP computation and a shared W8A8 head.
Prompts comprise 16 math, 8 Chinese explanation, and 8 English generation tasks.
Both target and MTP use cached decoding; acceptance includes the verifier bonus schedule.
Detailed records are in evaluation/mtp/.
mtp_coordinate_adapter.json records the external
alignment needed between the rotated target and the original MTP. For row-vector
states, rotation R, and original final RMSNorm gain gamma, the transforms are:
MTP embedding input = e_rot @ R.T
MTP hidden input = (h_postnorm_rot @ R.T) * gamma
Shared head input = (z_mtp / gamma) @ R
Limitations
- This C7 checkpoint does not provide native vLLM/SGLang MTP support. The adapter JSON is project metadata, not an adapter format those engines consume. The 81.98% result requires the external transforms above; retaining BF16 MTP weights alone gives the unaligned result. Standard Transformers text generation uses the ordinary autoregressive path.
- CUDA vLLM/SGLang execution of this packed W4A8 format is unverified. The checkpoint differs from W4A16, groupwise GPTQ/AWQ, GGUF, and QQQ's native kernel format.
- Full benchmark accuracy under stock compressed-tensors scaling and production throughput have not been measured. CMMLU/GSM8K scores are fixed subsets.
- Image/video accuracy, thinking mode, long context, and sampled or multi-candidate MTP behavior have not been evaluated.
License
Apache-2.0. Original model by the Qwen team, Alibaba Cloud; this is an independent
quantized derivative. Source file revisions are recorded in
source_provenance.json. See LICENSE and NOTICE.
ไธญๆ
ๆฌๆจกๅๆฏ Qwen3.5-4B ็ C7 W4A8/W8A8 ๆททๅ้ๅ็๏ผ้็จๅบๅฎ็ฆป็บฟๆฎๅทฎๆ่ฝฌ ไธ QQQ๏ผ197 ไธช W4A8 Linearใ52 ไธช W8A8 Linear๏ผdecoder Linear ๅๆฐไธญ 97.91% ไธบ W4ใ้่พๅบ้้ๅฏน็งฐๆ้ใFP32 scaleใ้ token ๅจๆ INT8 ๆฟๆดป๏ผ compressed-tensors ๆๅ ๏ผtensor ๅ ฑ 4.645 GB๏ผๅ ๅซ BF16 ่ง่งๅๆฏๅ MTPใ
ๅๅงๆ้ไป ModelScope ไธ่ฝฝ๏ผ็ๆฌไธบ fcb1a040bb418b0b8add6f6f6c475386abc2cb97ใ
W8 ่ๅดไธบ่พๅบๅคดใ24 ๅฑ็บฟๆงๆณจๆๅ็ in_proj_a/in_proj_b๏ผไปฅๅ็ฌฌ 1ใ14ใ28 ๅฑ
MLP down_projใๆฎๅทฎไฝฟ็จ seed 42 ็ signed Hadamard 20ร128 ๆ่ฝฌ๏ผไธปๅนฒๆ ๅจ็บฟๆ่ฝฌใ
ๆ กๅไฝฟ็จ 256 ๆกๅฎๆดๅฏน่ฏ๏ผไธญๆ 128ใ่ฑๆ 64ใๆฐๅญฆ 64๏ผๅ
ฑ 97,794 token๏ผๆ ๆชๆญใ
| ๆจกๅ | WikiText PPL โ | CMMLU /201 โ | GSM8K /100 โ | ๆฐๅญฆๆชๆญๆฐ โ |
|---|---|---|---|---|
| ๅๅงๆ้ | 9.944655 | 158 | 93 | 2 |
| C7 W4A8/W8A8 | 10.476089 | 151 | 95 | 1 |
PPL ็ธๅฏนไธๅ **5.34%**ใ็ปๆๆฅ่ช INT8รINT8โINT32 ๅ่ GEMM๏ผๆฟๆดปๆ full-K
้ token maxabs/127 ้ๅ๏ผ้็บฟๆงไธ KV ๆ FP32 ่ฎก็ฎ๏ผๅ
ณ้ญ TF32ใ
WikiText ไธบๅ
จ้จ 580 ไธช 512-token ๅใ147,900 ไธช่ฎกๅ token๏ผCMMLU ไธบ 67 ็งๅ 3 ้ข๏ผ
GSM8K ไธบๅบๅฎ 100 ้ข๏ผgreedy ๆๅค 512 ไธชๆฐ token๏ผๆชๆญ่ฎก้๏ผๅ
ณ้ญ thinkingใ
้้ข็ปๆใ็ฒพ็กฎไฝๅฎฝๆธ
ๅๅๆ้้ชๆถ่ฎฐๅฝๅทฒ้ไปๅบๆไพใ
ๆฎ้่ชๅๅฝ็ๆๅทฒ้่ฟๆ ๅ Transformers FP32 CPU ๅ ่ฝฝ้ชๆถ๏ผไพ่ตๅไปฃ็ ่งไธๆใ ๆ ๅ compressed-tensors ็้ป่ฎคๆฟๆดป็ผฉๆพไธไธ่ฟฐๅ่่ฏๆตๅฃๅพไธๅใ
MTP ็ 15 ไธชๅผ ้ไฟๆๆบๆจกๅ BF16 ๅๅผใ32 ๆกๅบๅฎๆ็คบ็ๅ่ๆฅๅ็๏ผๅๆจกๅ
**84.08%**๏ผC7 ๅ ๅๆ ้้
**81.98%**๏ผไธ้ 2.10 ไธช็พๅ็น๏ผไธๅ ้้
ไธบ **0.50%**ใ
ๅๆ ๅๆขๅๅๆฐ่งไธๆๅ mtp_coordinate_adapter.jsonใ
ๅฑ้๏ผๆฌ C7 ๆ้ๅฐไธๆฏๆ็ดๆฅ็จๅ็ vLLM/SGLang ๅผๅฏ MTP๏ผ่ฏฅ JSON ไธๆฏๅผๆ ๆฏๆ็้้ ๆ ผๅผ๏ผ81.98% ไธบๅ ๅซๅค้จๅๆ ๅๆข็ๆดๆฐๅ่็ปๆใ่ฏฅ packed W4A8 ๆ ผๅผ็ CUDA vLLM/SGLang ่ฟ่กใ็ไบงๅๅใๆ ๅ compressed-tensors ๅฎๆด็ฒพๅบฆใๅพๅ/่ง้ขใ thinkingใ้ฟไธไธๆๅ้ๆ ท MTP ๅฐๆช้ชๆถใ
- Downloads last month
- 147