Instructions to use ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16")
model = AutoModelForMultimodalLM.from_pretrained("ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16

SGLang

How to use ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16 with Docker Model Runner:
```
docker model run hf.co/ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16
```

gemma-4-A4B-98e-v7-coder — NVFP4A16 (vLLM)

NVFP4A16 quantization of ManniX-ITA/gemma-4-A4B-98e-v7-coder-it, the loop-fixed code prune of Gemma 4 26B-A4B (128→98 experts/layer, ~20.8B). 2 shards, ~13.4 GB — fits a single 16 GB GPU with room for KV-cache.

NVFP4A16 = 4-bit NVFP4 weights with FP8 (e4m3) block scales and bf16 activations (A16). Quantized with NVIDIA TensorRT-Model-Optimizer (modelopt) main (0.45.0.dev), whose _QuantFusedExperts plugin handles Gemma 4's fused MoE experts. Every exported weight/scale tensor is finiteness-verified before upload.

Serve with vLLM

python -m vllm.entrypoints.openai.api_server \
    --model ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16 \
    --served-model-name v7-coder \
    --port 8000 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 65536 \
    --max-num-batched-tokens 8192 \
    --dtype bfloat16 \
    --trust-remote-code \
    --reasoning-parser gemma4 \
    --default-chat-template-kwargs '{"enable_thinking": true}'

Gemma 4 gotchas (all required): --max-num-batched-tokens 8192 (the MM-encoder budget; the default 2048 < max_tokens_per_mm_item and crashes at boot), --max-model-len 65536 (templates may request up to ~49k gen tokens), and the gemma4 reasoning parser for the thinking format. NVFP4 needs a Blackwell/Hopper or Ada GPU on a recent vLLM.

Benchmarks

NVFP4A16 is a deployment format and is not separately benchmarked (cohort policy). The table is the cohort's same-host Q6_K · llama.cpp · greedy reference (temperature 0.0, top_p 1.0, top_k 0), read from summary.json; it is representative of this model's quality. Row-max in bold. This repo = v7-coder.

Benchmark	128e (unpruned)	v6-coder	v7-coder	v7-coderx
GPQA-diamond (198q)	67.17	61.11	51.52	51.01
AIME (30q)	73.33	56.67	80.00	76.67
MATH500 (100q)	92.00	89.00	95.00	95.00
GSM8K (100q)	89.00	88.00	91.00	93.00
ARC-Challenge (full)	96.50	95.39	92.15	86.60
IFEval (100q, strict)	97.00	92.00	92.00	92.00
HumanEval (164)	97.56	98.17	98.17	96.95
HumanEval+ (164)	92.07	92.68	92.07	93.29
LCB-medium-55 v4	96.36	92.73	98.18	92.73
LCB-medium-100 v4	97.00	94.00	94.00	91.00
MultiPL-E (100)	90.00	89.00	89.67	89.00

_{Metrics: GPQA & GSM8K = exact_match flexible-extract · MATH500 = math_verify ·
ARC & AIME = exact_match · IFEval = prompt_level_strict_acc · HumanEval/+ = pass@1
chat-extract · LCB-55/100 & MultiPL-E = pass@1. 128e uses the lcb_medium_55/100
templates; the prunes use lcb_medium_*_v4 (corrected harness, equivalent task). The
all-hard LCB-77 cross-model slice: 128e 79.22% / v7-coder 84.42% / v7-coderx 85.71%.}

Recipe (summary)

98e prune from 128e via the fkbroad code recipe (generate_drop_map_v5: generic_code 3×, targeted_lcb_medium_55 2×, all other targeting 0; target=98, protect_top=16, alpha=2.0, strategy=max, no per-layer floor clamp), then the agentic loop-protection force-keep (agentic_eog, 46 experts, 0/46 dropped — the loop fix that replaces fs2440), then the mandatory shared-FFN α=1.2 upweight, then NVFP4A16 quantization. No targeted_gpqa term and no DERN fold. Full recipe and the calibration-class table are on the bf16 card.

Intended use & limitations

A compact, vLLM-deployable Gemma 4 checkpoint for agentic coding and code reasoning — the balanced code member of the v7-coder cohort (leads LCB-medium + HumanEval; the code-maximal sibling v7-coderx leads the all-hard LCB-77 and HE+). A research prune, not an official Google release; generic_multilingual is de-weighted (0×) and graduate science (GPQA 51.52%) is a budget axis. For llama.cpp/CPU deployment use the GGUF repo.

Lineage

128e → (v4 → v5 → v6-coder code line) → v7 competence-map rebuild → fkbroad code3/lcb2 selection + agentic loop-protection force-keep = v7-coder (loop-fixed; supersedes fs2440) → NVFP4A16. Built and evaluated on the omnimergekit toolchain.