gemma-4-31B-it-qat-assistant-4bit

This repository contains Multi-Token Prediction (MTP) drafter weights split from google/gemma-4-31B-it-qat-q4_0-unquantized-assistant for use with mlx-vlm speculative decoding.

This is not a standalone chat or text-generation model. Load it as the draft model alongside a compatible Gemma 4 31B target checkpoint.

Use with mlx-vlm

uv run mlx_vlm.generate \
  --model google/gemma-4-31B-it \
  --draft-model mlx-community/gemma-4-31B-it-qat-assistant-4bit \
  --draft-kind mtp \
  --prompt "Describe this image." \
  --max-tokens 256

For local weights:

uv run mlx_vlm.generate \
  --model /path/to/target-model \
  --draft-model /path/to/gemma-4-31B-mtp \
  --draft-kind mtp \
  --prompt "Describe this image." \
  --max-tokens 256

Model Details

  • Model type: gemma4_assistant
  • Target architecture: Gemma 4 31B
  • Precision: 4bit
  • Runtime: MLX / mlx-vlm
  • Format: Safetensors with MLX-compatible config and tokenizer files

The stored tensors are 4bit MLX-compatible drafter weights.

Intended Use

Use this repo only as a speculative decoding drafter for compatible Gemma 4 31B checkpoints. The target model verifies drafted tokens, while this MTP model proposes candidate tokens per decoding step.

Limitations

This checkpoint requires runtime support for Gemma 4 MTP draft models in mlx-vlm. Standard standalone generation through generic Transformers APIs is not expected to work with this repository by itself.

Please refer to the upstream google/gemma-4-31B-it model card and license terms for model usage constraints.

Downloads last month
234
Safetensors
Model size
0.5B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mlx-community/gemma-4-31B-it-qat-assistant-4bit

Collection including mlx-community/gemma-4-31B-it-qat-assistant-4bit