Chat templates

Chat templates render a list of messages into the exact prompt format a chat model expects. They are Jinja2 strings stored on the tokenizer and applied via tokenizer.apply_chat_template(messages, ...).

Why they exist

Different chat models use different prompt formats:

• Llama 3 family uses <|start_header_id|>user<|end_header_id|>....
• Qwen 2.5 uses <|im_start|>user\n...<|im_end|>.
• Mistral uses [INST]...[/INST].

If the user code had to hand-format prompts for each model, the surface would be impossible to keep correct. Chat templates push that knowledge onto the tokenizer.

Where the code lives

File	Purpose
src/transformers/utils/chat_template_utils.py (26K LOC)	The Jinja2 environment, helpers, and built-in filters
src/transformers/utils/chat_parsing_utils.py (14K LOC)	Parses model output back into structured messages (tool calls)
src/transformers/tokenization_utils_base.py	apply_chat_template, chat_template attribute

Basic usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

Set add_generation_prompt=True so the template appends the assistant's start-of-turn marker, ready for model.generate.

Function calling and RAG

PR #30621 (June 2024) extended the template format with first-class support for tool calls and RAG documents. Templates can now consume:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": { ... JSON Schema ... },
        },
    }
]
documents = [
    {"title": "...", "text": "..."},
]
prompt = tok.apply_chat_template(messages, tools=tools, documents=documents, ...)

Templates that opt into the new variables render them according to the model's expected format (e.g., a JSON tool list followed by a special <tools> block, or an interleaved RAG context section). Templates that don't opt in silently ignore the kwargs.

The companion chat_parsing_utils.py is the inverse: when a model emits <tool_call>{"name": "...", "arguments": {...}}</tool_call> (or its model-specific equivalent), the parser turns the raw text into a structured Python object.

Multimodal chat templates

For image / audio / video chat, templates accept content as a list of typed parts:

messages = [
    {"role": "user", "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image", "image": "/path/to.png"},
    ]},
]
inputs = processor.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True, tokenize=True)

Multimodal templates emit <|image|> (or model-specific) placeholders that are resolved by the matching ProcessorMixin (see Processing).

Storing a template

Templates are stored on the tokenizer:

tok.chat_template = "..."
tok.save_pretrained("./out")    # writes chat_template.json (or embeds in tokenizer_config.json)
tok.push_to_hub("my-org/my-tokenizer")

The persistence format has evolved: older tokenizers stored the template inline in tokenizer_config.json; newer ones use chat_template.json so very long templates don't bloat the config.

Template syntax

Templates are Jinja2. The tokenizer adds these built-in objects/filters:

• messages — the input list.
• tools, documents — the new function-calling / RAG inputs.
• add_generation_prompt — Boolean.
• bos_token, eos_token, pad_token, … — special tokens from the tokenizer.
• raise_exception(message) — fail with a helpful error.

The reference docs (docs/source/en/chat_templating.md, chat_templating_writing.md, chat_templating_multimodal.md, chat_response_parsing.md) show real-world templates and idioms.

Testing

tests/test_tokenization_common.py covers chat-template round trips. Per-model templates are validated in tests/models/<arch>/test_tokenization_<arch>.py when present.

Integration points

• Tokenization — apply_chat_template lives on the tokenizer.
• Processing — multimodal processors expose the same method.
• CLI — transformers chat and transformers serve apply the template before forwarding.
• Serving — the OpenAI-compatible server uses it for POST /v1/chat/completions.

Entry points for modification

• Edit a template → tok.chat_template = "...", save, push.
• Extend the template grammar (new variable type) → src/transformers/utils/chat_template_utils.py. Tests in tests/utils/test_chat_template_utils.py.
• Tool-call response parsing → src/transformers/utils/chat_parsing_utils.py.