Troubleshooting Empty Tool_calls In Vllm Qwen3-14B Model Inference

Aug 5, 2025 by ADMIN 67 views

[Badcase]: vllm Qwen3-14B Model Inference Tool_calls is Empty

Introduction

This article addresses a specific issue encountered while using the vllm framework with the Qwen3-14B model. The problem arises during inference, where the tool_calls output is consistently empty, and the expected tool call information appears within the content instead. This behavior is observed not only in the Qwen3-14B model but also in the qwen2.5-32b-instruct model. This article will delve into the details of the issue, the environment setup, and potential causes, and explore possible solutions to this problem.

Problem Description

The primary issue is that when using the vllm framework to perform inference with the Qwen3-14B model, the tool_calls field in the output is consistently empty. Instead of being correctly parsed and placed in the tool_calls array, the information related to tool calls appears within the content field of the ChatMessage. This misplacement indicates a parsing or extraction failure within the vllm framework when handling the output from the Qwen3-14B model.

To illustrate this, consider the following example output:

ChatMessage(
    role='assistant',
    content='<think>\nOkay, let\'s try to figure out what\'s going on here. ... </think>\n\n{\n  "name": "final_answer",\n  "arguments": {\n    "answer": "2023年AI智能体技术已广泛应用于智能制造、企业运营及医疗领域。 ... "\n  }\n}',
    tool_calls=[],
    raw=ChatCompletion(id='chatcmpl-5e31812037fd4239abd4dc71c34d4e13', choices=[...], stop_reason=None),
    token_usage=TokenUsage(input_tokens=9098, output_tokens=461, total_tokens=9559)
)

In this output, you can see that the tool_calls list is empty (tool_calls=[]), but the content field contains a JSON object that should have been parsed as a tool call. This behavior suggests that the tool call parsing mechanism is not working as expected, leading to the incorrect placement of tool call information within the content.

This issue is not isolated to the Qwen3-14B model; it has also been observed with the qwen2.5-32b-instruct model. This consistency indicates a potential underlying problem in the configuration or the vllm framework's handling of these models, rather than a model-specific issue.

Environment Information

The environment in which this issue was encountered includes the following key components and configurations:

Deployment Script

The deployment script used for running the vllm inference server is as follows:

nohup python3 -m vllm.entrypoints.openai.api_server \
       --model="/work/share/weights/Qwen3-14B" \
       --trust-remote-code \
       --enforce-eager \
       --max-model-len 16384 \
       --tensor-parallel-size 1 \
       --enable-auto-tool-choice \
       --tool-call-parser hermes \
       --port 6666 \
       --gpu-memory-utilization 1 \
       --dtype bfloat16 \
       --cpu-offload-gb 12 \
       --chat-template /work/share/chengxiang/vllm/template.jinja \
       > "${ASCEND_PROCESS_LOG_PATH}/Qwen3-14B.log" 2>&1 &

Key Configuration Parameters

--model: Specifies the path to the Qwen3-14B model weights (/work/share/weights/Qwen3-14B).
--trust-remote-code: Allows the execution of code from the model repository, which is necessary for some models.
--enforce-eager: Enforces eager execution mode, which can help with debugging.
--max-model-len: Sets the maximum context length for the model (16384 tokens).
--tensor-parallel-size: Configures the number of GPUs to use for tensor parallelism (1 in this case).
--enable-auto-tool-choice: Enables the automatic selection of tools by the model.
--tool-call-parser hermes: Specifies the tool call parser to use (Hermes).
--port: Sets the port for the API server (6666).
--gpu-memory-utilization: Configures GPU memory utilization (1 for maximum utilization).
--dtype: Sets the data type to bfloat16 for mixed-precision training and inference.
--cpu-offload-gb: Specifies the amount of CPU memory to use for offloading (12 GB).
--chat-template: Provides a custom chat template for formatting inputs and outputs (/work/share/chengxiang/vllm/template.jinja).

Hardware

The deployment is running on a 910B Ascend AI processor, which is a key factor in the environment's capabilities and constraints. The specific hardware configuration can influence the performance and behavior of the model, making it crucial to consider in troubleshooting.

Software

vllm Framework: The vllm framework is used for efficient inference with large language models.
Python 3: The deployment script is executed using Python 3.

The software versions and dependencies can also play a role in the issue. Ensuring that all components are compatible and up-to-date is essential for stable operation.

Analysis of the Issue

The core problem is that the tool_calls field is empty, and the tool call information is embedded within the content. This behavior suggests a failure in the parsing or extraction of tool calls by the vllm framework. There are several potential causes for this issue:

Incorrect Tool Call Parsing

The --tool-call-parser hermes option in the deployment script specifies the Hermes parser. If this parser is not correctly configured or has issues with the specific output format of the Qwen3-14B model, it might fail to extract the tool calls. The Hermes parser is designed to identify and extract structured tool call information from the model's output, but it relies on a specific format to do so effectively.

Chat Template Mismatch

The --chat-template /work/share/chengxiang/vllm/template.jinja option uses a custom chat template. If this template is not correctly formatted for the Qwen3-14B model's expected input and output structure, it could lead to parsing issues. Chat templates define how the input prompts are formatted and how the model's responses are structured. An incorrect template can cause the model to generate output that the parser cannot recognize.

Model Output Format

The Qwen3-14B model might be generating tool call information in a format that is not fully compatible with the Hermes parser or the vllm framework's expectations. Language models can have variations in how they format tool calls, and if the parser is not adapted to these variations, it can fail to extract the necessary information. This issue might also be related to the specific prompt or interaction pattern used with the model.

Configuration Issues

Other configuration parameters, such as --enable-auto-tool-choice, might interact with the tool call parsing mechanism. If auto-tool-choice is not functioning correctly, it could affect how the model formats its output, leading to parsing failures. It's essential to ensure that all related configuration options are correctly set and compatible with the model and parser.

vllm Framework Bugs

It is also possible that there is a bug within the vllm framework itself that is causing the tool call parsing to fail. While vllm is designed for efficient inference, like any software, it can have bugs that affect its functionality. Keeping the framework up-to-date and checking for known issues can help identify and resolve such problems.

Steps Taken to Address the Issue

The reporter has already taken several steps to address the issue, indicating a thorough approach to troubleshooting:

Checked GitHub README

The reporter has reviewed the GitHub README for the Qwen3 model, which is a crucial first step in understanding the model's requirements and usage. The README often contains important information about installation, configuration, and known issues.

Checked Qwen Documentation

The reporter has also checked the official Qwen documentation, which provides detailed information about the model's architecture, capabilities, and usage guidelines. This documentation is a valuable resource for understanding the model's expected behavior and any specific requirements for tool call handling.

Checked Related Framework Documentation

The reporter has examined the documentation for the vllm framework and other related tools. This step is important for understanding how the framework handles tool calls and any configuration options that might affect parsing.

Searched Issues

The reporter has searched the Qwen3 GitHub issues to see if others have encountered similar problems. This search can help identify known issues and potential solutions or workarounds.

By taking these steps, the reporter has demonstrated a proactive approach to problem-solving and has exhausted common troubleshooting methods. This suggests that the issue might be more complex and require further investigation.

Potential Solutions and Further Investigation

Given the analysis and the steps already taken, here are several potential solutions and areas for further investigation:

Verify Chat Template

Ensure that the custom chat template (/work/share/chengxiang/vllm/template.jinja) is correctly formatted for the Qwen3-14B model and the Hermes parser. Review the template to ensure that it includes the necessary delimiters and formatting elements for tool calls. You might need to adjust the template to match the expected output structure of the model.

Test with Default Chat Template

Try running the inference without the custom chat template to see if the issue persists. If using the default template resolves the problem, it indicates that the custom template is the source of the issue. This can help isolate whether the problem lies in the custom configuration or the core parsing logic.

Experiment with Different Tool Call Parsers

If vllm supports other tool call parsers, try using a different parser to see if it correctly extracts the tool calls. This can help determine if the issue is specific to the Hermes parser or a more general parsing problem. Different parsers might have varying levels of compatibility with different model output formats.

Inspect Model Output

Examine the raw output from the Qwen3-14B model to understand how it formats tool call information. This can help identify any discrepancies between the model's output and the parser's expectations. You might need to log the raw output and analyze it manually to understand its structure.

Update vllm Framework

Ensure that you are using the latest version of the vllm framework. Newer versions often include bug fixes and improvements that might address the tool call parsing issue. Regularly updating the framework can help resolve compatibility problems and take advantage of the latest features.

Simplify the Prompt

Try simplifying the prompt to see if a complex prompt is causing the issue. A simpler prompt might produce a more straightforward output that the parser can handle correctly. This can help rule out any prompt-related issues.

Check vllm Configuration

Verify that all vllm configuration options related to tool calls are correctly set. Pay close attention to options like --enable-auto-tool-choice and any other parameters that might affect tool call handling. Incorrectly configured options can lead to parsing failures.

Debugging and Logging

Implement more detailed logging to capture the raw model output, the parser's input, and any error messages. This can provide valuable insights into the parsing process and help identify the exact point of failure. Debugging tools and logging can reveal specific issues that are not immediately apparent.

Community Support

Reach out to the vllm community or the Qwen model developers for support. They might have encountered similar issues and can provide guidance or solutions. Engaging with the community can also help identify whether the problem is a known issue or a new bug.

Submit a Detailed Issue

If the problem persists, consider submitting a detailed issue to the vllm or Qwen GitHub repository, including all relevant information about your environment, configuration, and the steps you have taken to troubleshoot. Providing comprehensive details can help developers understand the issue and provide targeted assistance.

Conclusion

The issue of empty tool_calls in the vllm framework when using the Qwen3-14B model is a significant challenge that requires a systematic approach to resolve. By carefully analyzing the environment, configuration, and model output, and by following the troubleshooting steps outlined above, it is possible to identify the root cause and implement a solution. The consistency of this issue across different models (Qwen3-14B and qwen2.5-32b-instruct) suggests a potential underlying problem in the vllm framework's tool call parsing mechanism or the chat template configuration. Further investigation and experimentation will be necessary to fully resolve this issue and ensure the correct extraction of tool call information.