Tokenizer

The tokenizer/ package implements the tokenizers used by ollamarunner and by the daemon's prompt-assembly path.

Purpose

Turn user input into token IDs, and model output back into characters, in a way that matches the source repository's tokenizer exactly. Mismatches cause subtly wrong outputs that are hard to debug, so the tokenizer code is meticulously tested.

Directory layout

tokenizer/
├── tokenizer.go         # interface + factory
├── bpe/                 # byte-pair encoding implementation
├── sentencepiece/       # sentencepiece implementation
├── ...                  # tokenizer-family specific code
└── *_test.go            # unit tests with golden vectors

Key abstractions

Symbol	Location	Purpose
Tokenizer interface	tokenizer/tokenizer.go	Encode(string) []int, Decode([]int) string, plus special-token accessors.
BPE implementation	tokenizer/bpe/	Multi-regex BPE with offset tracking.
Sentencepiece implementation	tokenizer/sentencepiece/	Unigram and BPE-style sentencepiece models.

How it works

The runner asks Tokenizer.Encode for prompt tokens, feeds them through the model, sampling produces output tokens, and Tokenizer.Decode runs incrementally as the streaming response is emitted. The streaming aspect matters: a multi-byte UTF-8 character may straddle two tokens, so the decoder maintains state across calls so partial bytes are buffered until a full character emerges.

Recent fixes

tokenizer: fix multi-regex BPE offset handling (#15844) from late April 2026 corrected an offset bug in BPE pre-tokenization where multiple regex patterns were applied in sequence. This is the kind of fix the unit tests exist to catch.

Integration points

• Used by ollamarunner and the runner-side /tokenize / /detokenize HTTP routes.
• Used by the daemon when it needs to count tokens (e.g., truncating message history to fit num_ctx).
• Used by convert/ when extracting the tokenizer from a HuggingFace checkpoint and writing it into GGUF metadata.

Entry points for modification

• New tokenizer family → add a sibling subdirectory implementing Tokenizer. Register it in tokenizer/tokenizer.go.
• Bug in an existing tokenizer → add a golden test vector first, then fix. Tokenizer behavior is shared across many models, so changes are high-risk.

Key source files

File	Purpose
tokenizer/tokenizer.go	Tokenizer interface and factory.
tokenizer/bpe/	BPE implementation.
tokenizer/sentencepiece/	Sentencepiece implementation.