Instructions to use google/gemma-4-26B-A4B-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use google/gemma-4-26B-A4B-it with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="google/gemma-4-26B-A4B-it") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("google/gemma-4-26B-A4B-it") model = AutoModelForMultimodalLM.from_pretrained("google/gemma-4-26B-A4B-it", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- HuggingChat
- Notebooks
- Google Colab
- Kaggle
- AMD Developer Cloud
- Local Apps Settings
- vLLM
How to use google/gemma-4-26B-A4B-it with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "google/gemma-4-26B-A4B-it" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "google/gemma-4-26B-A4B-it", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/google/gemma-4-26B-A4B-it
- SGLang
How to use google/gemma-4-26B-A4B-it with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "google/gemma-4-26B-A4B-it" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "google/gemma-4-26B-A4B-it", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "google/gemma-4-26B-A4B-it" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "google/gemma-4-26B-A4B-it", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use google/gemma-4-26B-A4B-it with Docker Model Runner:
docker model run hf.co/google/gemma-4-26B-A4B-it
Fix assistant content order before tool calls
An assistant message containing both a pre-tool explanation in content and tool_calls is currently rendered with the explanation after the tool call and its response. This changes the order when a generated OpenAI-style assistant message is added back to conversation history.
Move content rendering before tool calls and tool responses. Since that content now precedes the tool response, it must not close a final tool-response turn: leave that turn open for the assistant continuation, and close it when the next non-tool message is a user.
For this history:
messages = [
{"role": "user", "content": "Weather?"},
{
"role": "assistant", "content": "I'll check.",
"tool_calls": [{"id": "call_1", "type": "function", "function": {
"name": "get_weather", "arguments": {"city": "London"}
}}],
},
{"role": "tool", "tool_call_id": "call_1", "content": "sunny"},
]
prompt = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
assert prompt.index("I'll check.") < prompt.index("<|tool_call>")
assert prompt.endswith("<tool_response|>")
The model turn changes from:
<|turn>model
<|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|><|tool_response>response:get_weather{value:<|"|>sunny<|"|>}<tool_response|>I'll check.<turn|>
to:
<|turn>model
I'll check.<|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|><|tool_response>response:get_weather{value:<|"|>sunny<|"|>}<tool_response|>
Validation:
- With the actual Gemma 4 tokenizer and Transformers 5.5.4, the original template fails the content-order assertion above; the revised template passes. Appending a user message also produces the expected
<tool_response|><turn|>\n<|turn>user\nboundary. - Served
google/gemma-4-26B-A4B-itwith vLLM 0.19.1, TP=2 on two B300 GPUs,--enable-auto-tool-choice --tool-call-parser gemma4. Three consecutive Chat Completions rounds at temperature 0 returned nonempty pre-toolcontentand a structuredspeakcall. Each returned assistant message was appended as one message containing both fields. A subsequent streaming round emitted content before tool-call deltas.
This also addresses the nonempty-content turn-closure example reported in discussion #20. Its proposed template still renders the tool call before the content. The open #63/#66 scan optimizations and #67 thinking-channel change address separate behavior. Companion vLLM PR: vllm-project/vllm#66269.
AI assistance (OpenAI Codex) was used to implement the change and run validation.
Gemma 4 family rollout (2026-11-07)
The same content-order and premature-turn-closure bugs reproduce in all 23 standalone official Gemma 4 templates audited across 31B, 26B-A4B, 12B, E4B and E2B, including QAT/mobile variants. The templates fall into two text versions with the same affected logic.
Each minimally patched file passed content ordering, continuation after a tool result, and the following user-turn boundary checks on B300 with Transformers 5.5.4. These are serialization tests using the installed 26B-A4B tokenizer and an explicit template override for each repository. All published PR templates have been read back and matched to the validated files.
This rollout contains 22 PRs submitted for this fix, including this one. The existing 12B PR #12 already fixes this behavior through a broader template rewrite; it was independently verified and is reused for that repository. Its empty thought-channel suffix correctly remains inside the open model turn.
vLLM has one shared Gemma 4 example template, covered by vllm-project/vllm#66269.
| Repository | Pull request |
|---|---|
google/gemma-4-12B-it |
#12 (existing upstream fix) |
google/gemma-4-12B-it-qat-q4_0-unquantized |
#8 |
google/gemma-4-12B-it-qat-q4_0-unquantized-assistant |
#5 |
google/gemma-4-12B-it-qat-w4a16-ct |
#13 |
google/gemma-4-26B-A4B-it |
#73 |
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized |
#7 |
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant |
#7 |
google/gemma-4-31B-it |
#154 |
google/gemma-4-31B-it-qat-q4_0-unquantized |
#7 |
google/gemma-4-31B-it-qat-q4_0-unquantized-assistant |
#9 |
google/gemma-4-31B-it-qat-w4a16-ct |
#11 |
google/gemma-4-E2B-it |
#55 |
google/gemma-4-E2B-it-qat-mobile-ct |
#7 |
google/gemma-4-E2B-it-qat-mobile-transformers |
#9 |
google/gemma-4-E2B-it-qat-q4_0-unquantized |
#6 |
google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant |
#5 |
google/gemma-4-E2B-it-qat-w4a16-ct |
#6 |
google/gemma-4-E4B-it |
#60 |
google/gemma-4-E4B-it-qat-mobile-ct |
#6 |
google/gemma-4-E4B-it-qat-mobile-transformers |
#7 |
google/gemma-4-E4B-it-qat-q4_0-unquantized |
#6 |
google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant |
#6 |
google/gemma-4-E4B-it-qat-w4a16-ct |
#8 |
@MaartenGr @lucianommartins Could you help review this Gemma 4 chat-template fix?
When an assistant message contains both content and tool_calls, the template reorders its content after the tool call and response. This PR preserves the original sequence and keeps the model turn open for continuation after a tool result. The description includes a minimal reproduction and before/after validation.
The corresponding vLLM fix is waiting for upstream acceptance:
https://github.com/vllm-project/vllm/pull/66269
Since you worked on previous Gemma 4 template updates, could you review this or point me to the appropriate maintainer? Thanks!