Fix assistant content order before tool calls

#73
by youmuorz - opened

An assistant message containing both a pre-tool explanation in content and tool_calls is currently rendered with the explanation after the tool call and its response. This changes the order when a generated OpenAI-style assistant message is added back to conversation history.

Move content rendering before tool calls and tool responses. Since that content now precedes the tool response, it must not close a final tool-response turn: leave that turn open for the assistant continuation, and close it when the next non-tool message is a user.

For this history:

messages = [
    {"role": "user", "content": "Weather?"},
    {
        "role": "assistant", "content": "I'll check.",
        "tool_calls": [{"id": "call_1", "type": "function", "function": {
            "name": "get_weather", "arguments": {"city": "London"}
        }}],
    },
    {"role": "tool", "tool_call_id": "call_1", "content": "sunny"},
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
assert prompt.index("I'll check.") < prompt.index("<|tool_call>")
assert prompt.endswith("<tool_response|>")

The model turn changes from:

<|turn>model
<|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|><|tool_response>response:get_weather{value:<|"|>sunny<|"|>}<tool_response|>I'll check.<turn|>

to:

<|turn>model
I'll check.<|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|><|tool_response>response:get_weather{value:<|"|>sunny<|"|>}<tool_response|>

Validation:

  • With the actual Gemma 4 tokenizer and Transformers 5.5.4, the original template fails the content-order assertion above; the revised template passes. Appending a user message also produces the expected <tool_response|><turn|>\n<|turn>user\n boundary.
  • Served google/gemma-4-26B-A4B-it with vLLM 0.19.1, TP=2 on two B300 GPUs, --enable-auto-tool-choice --tool-call-parser gemma4. Three consecutive Chat Completions rounds at temperature 0 returned nonempty pre-tool content and a structured speak call. Each returned assistant message was appended as one message containing both fields. A subsequent streaming round emitted content before tool-call deltas.

This also addresses the nonempty-content turn-closure example reported in discussion #20. Its proposed template still renders the tool call before the content. The open #63/#66 scan optimizations and #67 thinking-channel change address separate behavior. Companion vLLM PR: vllm-project/vllm#66269.

AI assistance (OpenAI Codex) was used to implement the change and run validation.

Gemma 4 family rollout (2026-11-07)

The same content-order and premature-turn-closure bugs reproduce in all 23 standalone official Gemma 4 templates audited across 31B, 26B-A4B, 12B, E4B and E2B, including QAT/mobile variants. The templates fall into two text versions with the same affected logic.

Each minimally patched file passed content ordering, continuation after a tool result, and the following user-turn boundary checks on B300 with Transformers 5.5.4. These are serialization tests using the installed 26B-A4B tokenizer and an explicit template override for each repository. All published PR templates have been read back and matched to the validated files.

This rollout contains 22 PRs submitted for this fix, including this one. The existing 12B PR #12 already fixes this behavior through a broader template rewrite; it was independently verified and is reused for that repository. Its empty thought-channel suffix correctly remains inside the open model turn.

vLLM has one shared Gemma 4 example template, covered by vllm-project/vllm#66269.

Repository Pull request
google/gemma-4-12B-it #12 (existing upstream fix)
google/gemma-4-12B-it-qat-q4_0-unquantized #8
google/gemma-4-12B-it-qat-q4_0-unquantized-assistant #5
google/gemma-4-12B-it-qat-w4a16-ct #13
google/gemma-4-26B-A4B-it #73
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized #7
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant #7
google/gemma-4-31B-it #154
google/gemma-4-31B-it-qat-q4_0-unquantized #7
google/gemma-4-31B-it-qat-q4_0-unquantized-assistant #9
google/gemma-4-31B-it-qat-w4a16-ct #11
google/gemma-4-E2B-it #55
google/gemma-4-E2B-it-qat-mobile-ct #7
google/gemma-4-E2B-it-qat-mobile-transformers #9
google/gemma-4-E2B-it-qat-q4_0-unquantized #6
google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant #5
google/gemma-4-E2B-it-qat-w4a16-ct #6
google/gemma-4-E4B-it #60
google/gemma-4-E4B-it-qat-mobile-ct #6
google/gemma-4-E4B-it-qat-mobile-transformers #7
google/gemma-4-E4B-it-qat-q4_0-unquantized #6
google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant #6
google/gemma-4-E4B-it-qat-w4a16-ct #8

@MaartenGr @lucianommartins Could you help review this Gemma 4 chat-template fix?

When an assistant message contains both content and tool_calls, the template reorders its content after the tool call and response. This PR preserves the original sequence and keeps the model turn open for continuation after a tool result. The description includes a minimal reproduction and before/after validation.

The corresponding vLLM fix is waiting for upstream acceptance:
https://github.com/vllm-project/vllm/pull/66269

Since you worked on previous Gemma 4 template updates, could you review this or point me to the appropriate maintainer? Thanks!

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment