Qwen

/

Qwen3.5-397B-A17B

like 1.44k

Follow

Qwen 79.2k

Hi Qwen Team and community,
Qwen 团队和各位开发者好，

I encountered a severe crash issue when deploying this model using vLLM for Function Calling / Tool Calling.
我在使用 vLLM 部署该模型并进行工具调用（Function Calling）时遇到了一个严重的崩溃问题。

The Problem / 问题:
If the model hallucinates and outputs a non-JSON object for arguments (e.g., just a raw string like "action"), the client (like LiteLLM or OpenClaw) usually catches this and sends back a tool role message like "Validation failed for tool..." to trigger self-correction.
However, when vLLM tries to render this conversation history using the default chat_template, it crashes immediately with a 500/400 Bad Request error. The model never gets a chance to correct itself.
当大模型产生幻觉，没有按照 JSON Object 格式输出 arguments（比如直接输出了一个纯字符串 "action"）时，客户端（如 LiteLLM）通常会将这个错误记录作为历史上下文发回给模型，期望模型进行自我纠错（Self-Correction）。
但是，当 vLLM 尝试使用默认的 chat_template 渲染这段历史对话时，整个服务端会直接崩溃，抛出 500/400 Bad Request 错误，导致模型根本没机会看到报错信息。

Root Cause / 问题原因:
The bug is in the Jinja2 chat_template inside tokenizer_config.json. The template blindly assumes that tool_call.arguments is always a successfully parsed dict (Mapping).
问题出在 tokenizer_config.json 中的 chat_template。Jinja 模板代码强制假设 tool_call.arguments 永远是一个合法的字典（Mapping）。当接收到纯字符串时，调用 |items 过滤器会直接触发 Jinja2 的底层类型错误：TypeError: Can only get item pairs from a mapping.。

{# Defective code snippet in default template #}
{%- if tool_call.arguments is defined %}
    {%- for args_name, args_value in tool_call.arguments|items %}
        {{- '<parameter=' + args_name + '>\n' }}
        ...

When it receives a plain string (like "action"), the |items filter throws a native Jinja2 TypeError: Can only get item pairs from a mapping., terminating the entire request.

Solution / 修复方案
I have rewritten the arguments parsing logic in the chat_template to be highly robust. It now includes type sniffing and safe fallback mechanisms. If the model outputs garbage, the template wraps it in a special <parameter=raw_error_arguments> tag instead of crashing, allowing the LLM to see its mistake in the next turn.
我重写了 chat_template 中处理 arguments 的逻辑，提高了鲁棒性（包含类型嗅探和安全反序列化）。如果模型输出了无法解析的乱码或字符串，模板不再崩溃，而是将其安全地包裹在 <parameter=raw_error_arguments> 标签中，从而让大模型在下一轮能够感知并纠正自己的语法错误。

Robust Code Snippet (Replace the original arguments parsing loop):

{%- if tool_call.arguments is defined %}
    {%- set args = tool_call.arguments %}
    
    {# 1. Try safe JSON parsing if it's a string that looks like a dict / 如果是字符串且类似 JSON 字典，尝试安全解析 #}
    {%- if args is string %}
        {%- set str_args = args | trim %}
        {%- if str_args.startswith('{') and str_args.endswith('}') %}
            {%- set args = str_args | fromjson %}
        {%- endif %}
    {%- endif %}

    {# 2. Normal iteration if it's a valid dict / 如果是合法的字典，正常遍历 #}
    {%- if args is mapping %}
        {%- for args_name, args_value in args.items() %}
            {{- '<parameter=' + args_name + '>\n' }}
            {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
            {{- args_value }}
            {{- '\n</parameter>\n' }}
        {%- endfor %}
    {%- else %}
        {# 3. Ultimate Fallback: Output as raw parameter to prevent rendering crash / 容错：如果LLM生成了非 JSON，作为特殊 raw parameter 输出，防止引擎崩溃 #}
        {{- '<parameter=raw_error_arguments>\n' }}
        {{- args | string }}
        {{- '\n</parameter>\n' }}
    {%- endif %}
{%- endif %}

Extra Debug Log (For Reference) / 附加 LiteLLM （我用于管理各个LLM并提供统一出口）报错日志供参考：

litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.BadRequestError: Hosted_vllmException - {"error":{"message":"Can only get item pairs from a mapping.","type":"BadRequestError","param":null,"code":400}}. 
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/jinja2/filters.py", line 249, in do_items
    raise TypeError("Can only get item pairs from a mapping.")
TypeError: Can only get item pairs from a mapping.

Hope this helps improve the robustness of Qwen's tool calling capabilities in production!
希望这能帮助提升 Qwen3.5 在生产环境中工具调用的稳定性！