Tokenizer

Active contributors: Jong Wook Kim

Purpose

Wrap a tiktoken.Encoding with Whisper-specific special tokens (<|startoftranscript|>, <|nospeech|>, language tokens, task tokens, 1501 timestamp tokens) and provide cached property accessors plus helpers for building the start-of-transcript prefix and splitting tokens into words. Two encodings are shipped: gpt2 (English-only models) and multilingual (everything else).

Directory layout

whisper/
├── tokenizer.py
└── assets/
    ├── gpt2.tiktoken           # English-only BPE merges, ~836 KB
    └── multilingual.tiktoken   # multilingual BPE merges, ~817 KB

Key abstractions

Symbol	Description
LANGUAGES	OrderedDict-style dict of 99 ISO codes → human-readable names. Order is the language token order.
TO_LANGUAGE_CODE	Inverse of LANGUAGES, plus aliases (burmese→my, valencian→ca, etc.).
Tokenizer	Frozen-style dataclass wrapping tiktoken.Encoding with cached properties for special tokens.
get_encoding(name, num_languages)	Loads the on-disk BPE merges and constructs a tiktoken.Encoding with all Whisper specials. LRU-cached.
get_tokenizer(multilingual, *, num_languages=99, language=None, task=None)	Builds a Tokenizer. LRU-cached.

How it works

Loading the BPE merges

get_encoding(name) reads whisper/assets/{name}.tiktoken, which is a plain text file of <base64-bytes> <rank> lines:

ranks = {
    base64.b64decode(token): int(rank)
    for token, rank in (line.split() for line in open(vocab_path) if line)
}

It then builds the special-token list:

<|endoftext|>
<|startoftranscript|>
<|en|> <|zh|> ... <|<num_languages-1>>
<|translate|>
<|transcribe|>
<|startoflm|>
<|startofprev|>
<|nospeech|>
<|notimestamps|>
<|0.00|> <|0.02|> ... <|30.00|>   # 1501 tokens at 0.02 s resolution

Each special token gets the next sequential vocabulary slot. The tiktoken.Encoding's pat_str is the GPT-2 default regex, so non-special text tokenizes identically to GPT-2.

Vocabulary sizes

• English-only (gpt2): base GPT-2 merges plus 1 + 0 (no language tokens, since these models do not auto-detect) + 4 + 1501 specials = roughly 51,864.
• Multilingual: base merges plus 1 + 99 + 4 + 1501 = roughly 51,865 to 51,966 depending on num_languages.

Whisper.is_multilingual checks n_vocab >= 51865. Whisper.num_languages recovers the language count by subtracting the fixed offset.

Tokenizer construction

Tokenizer.__post_init__ derives the sot_sequence based on which language and task the user asked for:

sot_sequence = [<|startoftranscript|>]
if language is not None:
    sot_sequence.append(sot + 1 + langs.index(language))   # the matching <|<lang>|>
if task is not None:
    sot_sequence.append(<|transcribe|> if task == "transcribe" else <|translate|>)

For English-only models, language and task are kept None and the prefix is just [<|startoftranscript|>].

Cached properties

The class uses @cached_property for every special-token ID lookup so the dict access happens only once per token name:

eot, transcribe, translate, sot, sot_lm, sot_prev, no_speech, no_timestamps, timestamp_begin, language_token, all_language_tokens, all_language_codes, sot_sequence_including_notimestamps, non_speech_tokens.

timestamp_begin is the token ID of <|0.00|> (the first timestamp). All timestamp tokens are at IDs [timestamp_begin, timestamp_begin+1500]; decode() strips them by default. decode_with_timestamps keeps them, rendering as e.g. <|1.08|>.

non_speech_tokens

Returns a tuple of token IDs that mostly correspond to brackets, parentheses, music notes, and similar markers that should be suppressed at sampling time. Built by encoding each candidate symbol both bare and with a leading space, taking the first token where the encoding is exactly one token (or the symbol falls in the U+2640–267F miscellaneous-symbols range, where multi-byte tokens still share the leading byte). Always includes the lone - and ' after-space tokens to prevent tokens like - or ' from starting words spuriously.

This list is what the default suppress_tokens="-1" expands to in DecodingOptions — see Decoding.

Decoding

decode(token_ids) strips out timestamp tokens before calling tiktoken.decode. decode_with_timestamps(token_ids) keeps them, which is what whisper/transcribe.py and whisper/timing.py rely on internally.

Splitting into words

split_to_word_tokens(tokens) dispatches on language:

• For Chinese, Japanese, Thai, Lao, Burmese, and Cantonese (zh, ja, th, lo, my, yue), it calls split_tokens_on_unicode(tokens) because these languages don't separate words with spaces. The split happens at every position where decoded tokens form a valid Unicode point (it carefully walks the replacement character \ufffd to detect partial surrogates).
• For everything else, split_tokens_on_spaces(tokens) first calls split_tokens_on_unicode, then merges any subword that doesn't start with a space (and isn't a special token) onto the preceding word.

This dual strategy produces sane "words" for both spaced and non-spaced scripts.

Integration points

• Imports from: tiktoken only (no other internal imports).
• Imported by: whisper/__init__.py (does not re-export the tokenizer module, but get_tokenizer is reachable via whisper.tokenizer); whisper/decoding.py (Tokenizer, get_tokenizer); whisper/timing.py:Tokenizer; whisper/transcribe.py (LANGUAGES, TO_LANGUAGE_CODE, get_tokenizer).

Entry points for modification

• Adding a language: edit LANGUAGES. Note that the order is what determines the language token ID — appending is safer than reordering.
• Adding a special token: extend the specials list in get_encoding(). New tokens land at the next vocabulary slot.
• Custom suppression set: pass an explicit suppress_tokens=[...] into DecodingOptions. Building it from non_speech_tokens plus your own additions is the common pattern.
• Different word-split heuristic: subclass Tokenizer and override split_to_word_tokens. Word timestamps in Timing call this method.

See also: Decoding for how the SOT sequence is consumed, and Reference: configuration for the full set of language codes and aliases.