Tool calling and reasoning

Active contributors: Cyrus Leung, Aaron Pham, Russell Bryant.

Purpose

Modern chat models emit two cross-cutting structured signals:

• Tool calls — function invocations (OpenAI tools, Anthropic tools, MCP).
• Reasoning — internal chain-of-thought wrapped in tags like <think>…</think>.

vLLM treats both as parser plug-ins so the engine remains model-agnostic.

Tool parsers

Where they live

vllm/tool_parsers/
├── __init__.py            # ToolParserManager + builtin registrations
├── abstract_tool_parser.py# ToolParserBase
└── per-model parsers (qwen, deepseek, llama, gpt-oss, mistral, hermes, granite, ...)

How they are registered

@ToolParserManager.register_module(name=["my-model"])
class MyToolParser(ToolParserBase):
    def extract_tool_calls_streaming(self, ...): ...
    def extract_tool_calls(self, ...): ...

They are also discoverable via the vllm.general_plugins entry point group. The OpenAI server picks the parser based on the model's chat_template_format or the explicit --tool-call-parser flag.

What they do

Streaming: as tokens arrive, the parser emits delta events (tool_calls.function.arguments chunks) so the OpenAI streaming response remains valid SSE.

Non-streaming: at the end of the response, the parser splits content from tool_calls and produces structured arguments.

Reasoning parsers

Where they live

vllm/reasoning/
├── __init__.py            # ReasoningParserManager
├── reasoning_parser.py    # base class
└── per-model parsers (deepseek_r1, qwen3, glm_45, gpt_oss, granite, hunyuan_a13b, ...)

Behavior

Reasoning parsers strip <think>...</think> (or similar) blocks from the user-visible text and route them into the reasoning_content field of the OpenAI response. They also signal reasoning_ended to the engine — the structured-output sampler uses that to gate grammar masking until reasoning is done.

vllm/v1/sample/thinking_budget_state.py enforces min_reasoning_tokens / max_reasoning_tokens from SamplingParams.reasoning_budget so models can't ramble forever in the reasoning span.

OpenAI integration

The OpenAI chat completion handler (vllm/entrypoints/openai/chat_completion/serving.py) wires both parsers together:

1. Look up the tool parser by model / flag (or fall back to no tools).
2. Look up the reasoning parser similarly.
3. While streaming output, feed tokens through both parsers.
4. Emit delta.tool_calls, delta.content, and delta.reasoning_content events to the client.

Anthropic, MCP, and gRPC frontends use the same parsers via shared helpers in vllm/entrypoints/openai/parser/.

Key source files

File	Purpose
vllm/tool_parsers/abstract_tool_parser.py	Base class
vllm/tool_parsers/__init__.py	ToolParserManager + builtin registrations
vllm/reasoning/__init__.py	ReasoningParserManager
vllm/reasoning/reasoning_parser.py	Base class
vllm/v1/sample/thinking_budget_state.py	Reasoning budget enforcement
vllm/entrypoints/openai/chat_completion/serving.py	Where both parsers are wired into chat completions

Entry points for modification

• New tool parser: subclass ToolParserBase, register with ToolParserManager.register_module(name=["model-id"]) (or via plug-in entry point).
• New reasoning parser: same pattern under vllm/reasoning/.
• Reasoning budget plumbing: respect the reasoning_budget field on SamplingParams and emit reasoning_ended=True once the reasoning span ends.