Gemma-4-26B-A4B-it-NVFP4

NVFP4 quantization of google/gemma-4-26b-a4b-it — the Mixture-of-Experts variant of Gemma 4 with 25.2B total parameters and ~3.8B active per token. Quantized using NVIDIA TensorRT Model Optimizer (modelopt) on DGX Spark hardware.

Supports text, image, and multimodal inference with a 256K token context window.

Base Model vs. NVFP4 Comparison

	Original (BF16)	NVFP4 (this)
Size on disk	~49 GB	~16 GB
Compression	1x	3.0x
Total parameters	25.2B	25.2B
Active parameters/token	~3.8B	~3.8B
Architecture	MoE: 128 experts, top-8 routing	Same
Context window	262,144 tokens	262,144 tokens
Modalities	Text + Image	Text + Image
Min GPU Memory	~50 GB+	~18 GB
DGX Spark (128 GB)	Runs, memory-constrained	Runs comfortably

Quantization Details

Method	NVFP4 (4-bit floating point weights)
Tool	NVIDIA modelopt
Group Size	16
Format	SafeTensors (3 shards)

Quality-Preserving Exclusions

The following layers are kept at full bfloat16 precision to preserve output quality:

Excluded Layer	Reason
MoE routers (all 30 layers)	Routing decisions are critical for MoE quality — quantizing routers degrades expert selection
Vision tower & embeddings	Image understanding quality degrades significantly under quantization
LM head	Final output layer directly impacts token probability accuracy

This is standard best practice for MoE quantization: compress the bulk of the transformer (attention + FFN layers) where FP4 has minimal quality impact, but keep sensitive routing, vision, and output layers at full precision. The tradeoff is a slightly larger model (~16 GB vs theoretical ~12 GB for full FP4) but significantly better quality retention.

MoE Quantization Challenge

Gemma 4 MoE stores expert weights as fused 3D tensors (shape [128, dim, dim]) rather than individual linear modules. NVIDIA modelopt only quantizes linear layers — the fused expert parameters (which are ~91% of the model) require a custom plugin to unfuse into individual linear layers before quantization.

Hardware

Quantized and serving on NVIDIA DGX Spark:

• GB10 Grace Blackwell processor
• 128 GB unified CPU+GPU memory
• CUDA 13.1
• SM 12.1

Model loads at ~16 GB, leaving substantial headroom for long-context inference with KV cache.

Serving with vLLM

Quick Start

vllm serve CyberFitz/gemma-4-26B-A4B-it-NVFP4 \
  --quantization modelopt \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.85 \
  --moe-backend marlin \
  --trust-remote-code

Key Flags

Flag	Why
--quantization modelopt	NVFP4 checkpoint format
--moe-backend marlin	Marlin kernel for efficient MoE expert layers
--kv-cache-dtype fp8	Saves memory for longer contexts
--trust-remote-code	Required for Gemma 4 architecture

Docker (DGX Spark)

docker run -d \
  --name gemma4-nvfp4 \
  --gpus all --ipc=host --network host \
  -e VLLM_NVFP4_GEMM_BACKEND=marlin \
  -v /path/to/model:/model \
  <your-vllm-image> \
  vllm serve /model \
    --quantization modelopt \
    --max-model-len 262144 \
    --kv-cache-dtype fp8 \
    --moe-backend marlin \
    --trust-remote-code

Testing

# Text generation
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-26B-A4B-it-NVFP4",
    "messages": [{"role": "user", "content": "Explain mixture of experts in one paragraph."}],
    "max_tokens": 200
  }'

Usage with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "CyberFitz/gemma-4-26B-A4B-it-NVFP4",
    device_map="auto",
    torch_dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained("CyberFitz/gemma-4-26B-A4B-it-NVFP4")

messages = [{"role": "user", "content": "Hello, how are you?"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Benchmarks

Coming soon — GSM8K, IFEval, and throughput benchmarks comparing BF16 vs NVFP4 on DGX Spark.

License

This model inherits the Gemma license from the base model.

Credits

Quantized by CyberFitz on NVIDIA DGX Spark hardware.