openai/whisper
Glossary
Terms that come up across the codebase.
| Term | Meaning |
|---|---|
| ASR | Automatic Speech Recognition. Whisper handles ASR plus translation to English. |
| mel spectrogram / log-mel | Time–frequency representation of audio used as model input. 16 kHz audio → STFT (N_FFT=400, HOP_LENGTH=160) → power → mel filterbank (80 or 128 channels) → log10 → normalized to roughly [-1, 1]. Computed in whisper/audio.py:log_mel_spectrogram. |
| n_mels | Number of mel filter channels. 80 for all models except large-v3 and turbo, which use 128. |
| n_audio_ctx | Encoder context length in mel frames after the strided conv stack. 1500 for all current models (= 30 s × 100 frames/s ÷ stride 2). |
| n_text_ctx | Decoder context length in tokens. 448 for all current models. |
| alignment heads | The subset of decoder cross-attention heads whose attention pattern correlates with audio↔text alignment. Used to compute word timestamps. Per-model masks are stored as base85+gzip in _ALIGNMENT_HEADS in whisper/__init__.py. |
SOT / <|startoftranscript|> |
The first token of the decoder input prefix. Tokens after it specify language and task. |
| sot_sequence | The full decoder prefix that primes the model: [<|startoftranscript|>, <|lang|>, <|task|>] for multilingual models, or just [<|startoftranscript|>] for English-only ones. Built in Tokenizer.__post_init__. |
| task | Either transcribe (X→X) or translate (X→English). Selected via the <|transcribe|> / <|translate|> token after the language token. |
| timestamp tokens | Special tokens <|0.00|>, <|0.02|>, …, <|30.00|> (1501 of them). The model emits these to delimit segment start and end. Resolution is 0.02 s. |
<|notimestamps|> |
A special token that disables timestamp emission for the rest of the decode call. |
<|nospeech|> |
A special token whose probability at the SOT position estimates the chance that the chunk contains no speech. Used by no_speech_threshold. |
| temperature fallback | The strategy in transcribe() of retrying decoding at successively higher temperatures (default 0.0, 0.2, 0.4, 0.6, 0.8, 1.0) when a decode looks bad (e.g. compression ratio too high or log-prob too low). |
| compression ratio | len(text_bytes) / len(zlib.compress(text_bytes)), computed by whisper.utils.compression_ratio. High values indicate repetitive output and trigger fallback. |
| logprob_threshold | Average per-token log-probability below which the decoder is considered to have failed; triggers fallback unless no_speech_prob is high enough to consider the segment silent. |
| no_speech_threshold | Probability threshold on <|nospeech|> above which a segment is skipped as silence (subject to logprob_threshold logic). |
| carry_initial_prompt | If true, transcribe() prepends the user-supplied initial prompt to every internal decode() call instead of letting it scroll out of the prompt window. |
| condition_on_previous_text | If true, the previous segment's tokens are passed as prompt to the next decode call. Disabling reduces inter-segment consistency but makes failure loops less sticky. |
| clip_timestamps | A list of start,end,start,end,… seconds restricting transcription to selected clips inside a longer file. |
| DTW | Dynamic Time Warping. Used in whisper/timing.py to align text tokens with audio frames for word-level timestamps. CPU implementation in Numba; GPU kernel in Triton. |
| KV cache | Cached key and value projections from previous autoregressive steps so the decoder only runs the latest token through self-attention. Installed via forward hooks in Whisper.install_kv_cache_hooks. |
| SDPA | PyTorch's torch.nn.functional.scaled_dot_product_attention. Whisper uses it when available; otherwise it computes attention manually. Disabled during DTW alignment so attention weights can be observed. |
load_audio |
Helper that shells out to ffmpeg to decode any input file to mono 16 kHz float32 PCM. |
pad_or_trim |
Pads with zeros or truncates to exactly 30 s (N_SAMPLES samples / N_FRAMES mel frames). Required because the encoder expects fixed-length input. |
| fallback | See temperature fallback. Implemented in decode_with_fallback inside whisper/transcribe.py. |
| hallucination silence threshold | When word timestamps are enabled, segments classified as anomalous and surrounded by silence are skipped, with the seek pointer moved past the silence. See transcribe() for the heuristics. |
| patience | Beam-search patience parameter (arxiv:2204.05424). With beam_size=B and patience=p, the decoder collects up to round(B*p) candidates before stopping. |
| length_penalty | Optional Google-NMT-style length penalty α used by the MaximumLikelihoodRanker to score completed candidates. None falls back to plain length normalization. |
| English-only / multilingual | English-only models are trained on English audio and use the GPT-2 BPE tokenizer; multilingual models use a 99-language vocabulary. Whisper.is_multilingual checks n_vocab >= 51865. |
tiktoken |
OpenAI's BPE library. Whisper uses tiktoken.Encoding directly, not the convenience helpers, so it can attach Whisper-specific special tokens. |
| EnglishTextNormalizer | Post-processing normalizer for evaluation: lowercases, expands contractions, normalizes spelling and number words, etc. Implemented in whisper/normalizers/english.py. Not used during normal transcription — it is for benchmark scoring. |
Built by Factory AutoWiki from public repository content. It is a generated preview for codebase exploration, not source-maintained documentation.