Decoding

Active contributors: Jong Wook Kim

Purpose

Run autoregressive decoding on a single 30 s audio chunk. Given a Whisper model and a Mel spectrogram, produce a list of DecodingResults — one per audio in the (optional) batch — each containing the predicted text, tokens, detected language, average log-probability, no-speech probability, and the compression ratio. This is the layer the long-form transcribe() calls many times per file.

Directory layout

whisper/
└── decoding.py

Key abstractions

Symbol	Description
DecodingOptions	Frozen dataclass of decoder configuration: task, language, sampling, prompt, suppress, fp16.
DecodingResult	Frozen dataclass of decoder output: text, tokens, language, probs, log-probs, ratios.
Inference	Interface for "given tokens and audio features, return next-token logits."
PyTorchInference	Concrete Inference that owns the KV cache and runs model.decoder per step.
SequenceRanker / MaximumLikelihoodRanker	Pick the best candidate per audio at the end of beam search / best-of-N sampling.
TokenDecoder	Strategy for picking the next token: GreedyDecoder or BeamSearchDecoder.
LogitFilter family	SuppressBlank, SuppressTokens, ApplyTimestampRules. In-place logit modifications.
DecodingTask	The orchestrator. Builds initial tokens, instantiates inference/decoder/filters, runs main loop.
detect_language(model, mel, tokenizer=None)	Special single-step decode that masks all non-language tokens. Returns top language token + per-language probabilities.
decode(model, mel, options=DecodingOptions(), **kwargs)	The public entry point. Bound onto Whisper.decode.

How it works

graph TD
    options[DecodingOptions] --> task[DecodingTask.__init__]
    model[Whisper] --> task
    task --> tok[Tokenizer for language/task]
    task --> inf[PyTorchInference]
    task --> dec{decoder}
    task --> filters[LogitFilter list:<br/>SuppressBlank<br/>SuppressTokens<br/>ApplyTimestampRules]
    dec -- temperature == 0 and beam_size --> beam[BeamSearchDecoder]
    dec -- otherwise --> greedy[GreedyDecoder]

    mel[mel: B x n_mels x 3000] --> run[DecodingTask.run]
    run --> feats[encoder forward]
    feats --> langd{language is None?}
    langd -- yes --> dl[_detect_language] --> rewrite[overwrite lang token in tokens]
    langd -- no --> loop
    rewrite --> loop
    loop[_main_loop] -->|step k| inf
    inf -->|logits| filters
    filters --> dec
    dec -->|completed?| loop
    loop --> finalize[decoder.finalize] --> rank[MaximumLikelihoodRanker.rank]
    rank --> result[List of DecodingResult]

Initial tokens

_get_initial_tokens builds the decoder prefix:

[<|startofprev|>, ...prompt..., <|startoftranscript|>, <|lang|>, <|task|>, ...prefix..., <|notimestamps|>?]

• prompt (text or token IDs) — previous-context priming. Truncated to fit n_ctx // 2 - 1 tokens.
• prefix (text or token IDs) — primes the current segment; preserved at decode time. Truncated to fit n_ctx // 2 - sample_len.
• The sot_sequence already includes <|startoftranscript|> plus the language and task tokens (built in Tokenizer.__post_init__).
• If options.without_timestamps, <|notimestamps|> is appended.

sample_begin is set to the length of this prefix and is used everywhere downstream to know where "sampled" tokens start.

Inference and KV caching

PyTorchInference.logits(tokens, audio_features) is the only place that calls model.decoder. It lazy-installs the KV cache hooks the first time it is called. After the first call it slices tokens[:, -1:] so only the newest token runs through self-attention; the cache returns the previously computed K/V for older tokens.

rearrange_kv_cache(source_indices) reorders cached K/V tensors when beam search reshuffles its beams. cleanup_caching() removes the forward hooks and clears the cache; it runs in a finally in _main_loop.

Logit filters

Three filters are composed in this order:

1. SuppressBlank: at the very first sampled position, masks out the space token and EOT so the model cannot emit an empty output.
2. SuppressTokens: masks any tokens listed in options.suppress_tokens. The default "-1" expands to tokenizer.non_speech_tokens (parens, brackets, music notes, etc.) plus the special tokens <|transcribe|>, <|translate|>, <|startoftranscript|>, <|startofprev|>, <|startoflm|>, and <|nospeech|>.
3. ApplyTimestampRules (only when timestamps are enabled): enforces three rules on the output:
   • <|notimestamps|> is always masked (it would only confuse the rest of the loop).
   • Timestamps must come in pairs (start, end). After a single timestamp, the next must be non-timestamp; after a pair, the next must be a timestamp again.
   • Timestamps must be non-decreasing within a segment, and at least one frame apart (no zero-length segments).
   • At sample_begin, only timestamp tokens are allowed (segments must start with a timestamp), bounded above by max_initial_timestamp_index.
   • At every step, if the total probability mass on timestamp tokens exceeds the largest non-timestamp probability, all non-timestamp logits are masked. This biases generation toward timestamps when the model is "uncertain" between text and a segment break.

Token decoders

• GreedyDecoder (temperature == 0): picks argmax. With temperature > 0, samples from Categorical(logits / temperature).
• BeamSearchDecoder (beam_size is not None, temperature == 0): standard beam search with patience. For each beam it considers beam_size + 1 candidate next tokens (so the EOT can pop out without losing any actual text continuations), keeps the top beam_size non-EOT and routes EOT-completing sequences into per-audio finished sets. Stops when each audio has round(beam_size * patience) finished candidates. rearrange_kv_cache keeps the inference cache in sync with the surviving beams.

_detect_language

When options.language is None (or options.task == "lang_id"), a single forward pass at sot_index + 1 is taken to score every language token. The detected language token is written into tokens[:, sot_index + 1] for the rest of the loop. This is also reachable as whisper.detect_language(model, mel).

Main loop

for i in range(self.sample_len):
    logits = self.inference.logits(tokens, audio_features)
    if i == 0 and self.tokenizer.no_speech is not None:
        # save no_speech_prob from the SOT-position logits
        probs_at_sot = logits[:, self.sot_index].float().softmax(dim=-1)
        no_speech_probs = probs_at_sot[:, self.tokenizer.no_speech].tolist()
    logits = logits[:, -1]
    for f in self.logit_filters: f.apply(logits, tokens)
    tokens, completed = self.decoder.update(tokens, logits, sum_logprobs)
    if completed or tokens.shape[-1] > self.n_ctx: break

no_speech_prob is computed exactly once, from the SOT-position logits at step 0, before any logit filter runs. This is what transcribe() later compares against no_speech_threshold.

After the loop, decoder.finalize(tokens, sum_logprobs) returns full token sequences (padded with EOT for greedy, or unfinished beams promoted to finished for beam search). MaximumLikelihoodRanker selects the best candidate per audio using either plain length normalization (length_penalty=None) or the Google NMT length penalty.

Verification

_verify_options rejects nonsense combinations: beam_size and best_of are mutually exclusive; best_of is incompatible with greedy sampling; patience requires beam_size; length_penalty must be in [0, 1].

Integration points

• Imports from: whisper/audio.py:CHUNK_LENGTH; whisper/tokenizer.py (Tokenizer, get_tokenizer); whisper/utils.py:compression_ratio.
• Imported by: whisper/__init__.py (re-exports DecodingOptions, DecodingResult, decode, detect_language); whisper/model.py (binds decode and detect_language onto Whisper); whisper/transcribe.py (calls model.decode).

Entry points for modification

• New sampling strategies: subclass TokenDecoder. Add a branch in DecodingTask.__init__.
• New logit constraints: subclass LogitFilter and add to the logit_filters list.
• New decoding options: extend DecodingOptions, then thread the field through _get_initial_tokens / _get_suppress_tokens / _main_loop as appropriate.
• Custom inference backend (e.g. ONNX, CT2): subclass Inference and instantiate it in DecodingTask.__init__ instead of PyTorchInference.

See also: Transcribe for the surrounding fallback / segment loop, and Tokenizer for what sot_sequence and non_speech_tokens actually contain.