Tokenization

Purpose

Tokenizers convert text to and from sequences of integer ids. The library supports three backends — pure Python (slow), Rust-backed tokenizers (fast), and mistral-common — under a single base class so user code stays the same.

Key abstractions

Class	File	Role
PreTrainedTokenizerBase	src/transformers/tokenization_utils_base.py (3,580 LOC)	Common API: encode/decode, padding, truncation, chat templates, Hub I/O
PreTrainedTokenizer (slow)	src/transformers/tokenization_python.py (2,393 LOC)	Pure-Python tokenizer base
PreTrainedTokenizerFast	src/transformers/tokenization_utils_tokenizers.py (66K LOC)	Wraps the Rust tokenizers library
MistralCommonTokenizer	src/transformers/tokenization_mistral_common.py (80K LOC)	mistral-common backend
SentencePieceTokenizer	src/transformers/tokenization_utils_sentencepiece.py	SentencePiece base
<Arch>Tokenizer, <Arch>TokenizerFast	src/transformers/models/<arch>/tokenization_<arch>.py	Per-architecture tokenizer
AutoTokenizer	src/transformers/models/auto/tokenization_auto.py	Late-binding factory
convert_slow_tokenizer	src/transformers/convert_slow_tokenizer.py (78K LOC)	Slow → fast conversion

The tokenizer triplet

A typical model directory contains:

• tokenization_<name>.py — the slow tokenizer (often inherits from SentencePieceTokenizer or hand-implements vocab logic).
• tokenization_<name>_fast.py — the fast tokenizer (inherits from PreTrainedTokenizerFast).

The library has 99 tokenization files. Many models share a tokenizer with another (e.g., GPT-2 family) and re-export it.

Common API

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
ids = tok("Hello world", return_tensors="pt")
print(tok.decode(ids["input_ids"][0]))

# Batch + padding
batch = tok(["a", "much longer sequence"], padding=True, truncation=True, max_length=64, return_tensors="pt")

# Chat template
chat = [{"role": "user", "content": "hi"}]
prompt_ids = tok.apply_chat_template(chat, return_tensors="pt", add_generation_prompt=True)

The base class implements __call__, encode, decode, batch_encode_plus, batch_decode, pad, truncate, apply_chat_template, from_pretrained, save_pretrained, push_to_hub.

Special tokens

Every tokenizer exposes pad_token, eos_token, bos_token, unk_token, sep_token, cls_token, mask_token, additional_special_tokens. Special tokens are tracked by SpecialTokensMixin (in tokenization_utils_base.py) and serialized into special_tokens_map.json.

Chat templates

The chat-template machinery lives in src/transformers/utils/chat_template_utils.py (26K LOC) and is exposed via tokenizer.chat_template and tokenizer.apply_chat_template. Templates are Jinja2 strings stored on the tokenizer alongside the vocab.

Chat templates support tool-calling and RAG since PR #30621 (June 2024). See Chat templates.

Slow → fast conversion

src/transformers/convert_slow_tokenizer.py (78K LOC) generates a fast tokenizers.Tokenizer from a slow tokenizer's vocab and merges. CLI helper: src/transformers/convert_slow_tokenizers_checkpoints_to_fast.py. The conversion is run for some models at training time so the fast version is available without re-encoding.

How loading works

graph LR
    User[from_pretrained] --> CFG{tokenizer_config.json}
    CFG -->|tokenizer_class| AutoTok[AutoTokenizer]
    AutoTok --> SlowOrFast{Fast available?}
    SlowOrFast -->|yes| Fast[<Arch>TokenizerFast]
    SlowOrFast -->|no| Slow[<Arch>Tokenizer]
    Fast --> Tokens[(tokenizer.json)]
    Slow --> Vocab[(vocab.json + merges.txt)]

AutoTokenizer.from_pretrained reads tokenizer_config.json, falls back to the tokenizer_class field, and uses TOKENIZER_MAPPING from tokenization_auto.py to pick the right pair.

Testing

• tests/test_tokenization_common.py (132,908 LOC) provides TokenizerTesterMixin.
• tests/test_tokenizers_backend_mixin.py, tests/test_sentencepiece_backend_mixin.py, tests/test_tokenization_mistral_common.py cover backend-specific behaviour.

Integration points

• The Hub I/O layer (src/transformers/utils/hub.py) handles vocab/merges/tokenizer.json.
• Pipelines that consume text (text-generation, text-classification, fill-mask, token-classification, question-answering, summarization, translation) instantiate the tokenizer via AutoTokenizer.
• Trainer uses tokenizers indirectly through the data collator.
• transformers chat and transformers serve apply chat templates before forwarding prompts.

Entry points for modification

• Add a tokenizer for a new model: drop a tokenization_<name>.py (and optionally tokenization_<name>_fast.py) into the model dir, register in tokenization_auto.py, run make fix-repo.
• Add a chat template: store the Jinja2 string at tokenizer.chat_template = "..." and re-save. Templates can also be pushed as part of the tokenizer files via push_to_hub.
• To add a new tokenizer backend, subclass PreTrainedTokenizerBase and implement the encode/decode/save contract; the existing tokenization_python.py, tokenization_utils_tokenizers.py, and tokenization_mistral_common.py are reference implementations.