Architecture

Whisper is a single-model, encoder-decoder Transformer wrapped in a thin Python pipeline. Audio comes in, a log-Mel spectrogram is computed, the encoder turns it into audio features, the decoder autoregressively generates text tokens (with optional language and task prefix tokens), and a small amount of glue code on top handles long files, fallbacks, and word timing.

High-level flow

graph LR
    A[audio file or array] -->|ffmpeg, resample| B[16 kHz mono PCM]
    B -->|STFT + mel filters| C[log-Mel spectrogram]
    C -->|sliding 30 s window| D[AudioEncoder]
    D -->|audio features| E[TextDecoder]
    E -->|token logits + KV cache| F[TokenDecoder<br/>greedy or beam]
    F -->|sampled tokens| E
    F -->|finished tokens| G[Tokenizer.decode]
    G -->|text + timestamps| H[Result writer<br/>txt / vtt / srt / tsv / json]
    E -.->|cross-attention QKs| I[DTW alignment<br/>word timestamps]
    I --> H

Module map

graph TD
    subgraph CLI
        cli[whisper/__main__.py<br/>whisper/transcribe.py:cli]
    end
    subgraph Pipeline
        transcribe[whisper/transcribe.py<br/>transcribe&#40;&#41;]
        decoding[whisper/decoding.py<br/>DecodingTask]
        timing[whisper/timing.py<br/>add_word_timestamps]
    end
    subgraph Model
        loader[whisper/__init__.py<br/>load_model]
        model[whisper/model.py<br/>Whisper, AudioEncoder, TextDecoder]
        triton[whisper/triton_ops.py<br/>DTW + median CUDA kernels]
    end
    subgraph Preproc
        audio[whisper/audio.py<br/>load_audio, log_mel_spectrogram]
        tok[whisper/tokenizer.py<br/>Tokenizer, get_tokenizer]
    end
    subgraph Postproc
        utils[whisper/utils.py<br/>WriteVTT/SRT/TSV/JSON]
        norm[whisper/normalizers/<br/>EnglishTextNormalizer]
    end

    cli --> transcribe
    transcribe --> audio
    transcribe --> decoding
    transcribe --> timing
    transcribe --> tok
    transcribe --> utils
    decoding --> model
    decoding --> tok
    timing --> model
    timing --> triton
    loader --> model

Two-level decoding

Whisper has two layers of "decoding" that are easy to confuse:

• Window decoding (whisper/decoding.py, DecodingTask): runs one forward pass per autoregressive step over a single 30 s mel window, with KV caching, beam search or greedy sampling, and logit filters that suppress non-speech tokens and enforce timestamp pairing.
• File-level transcription (whisper/transcribe.py, transcribe()): walks the whole file in 30 s chunks. For each chunk, it calls model.decode() (i.e. DecodingTask.run) inside a decode_with_fallback loop that retries with increasing temperature when results look bad. It stitches the segments together, advances seek based on the predicted timestamp tokens, and optionally calls add_word_timestamps() for word-level alignment.

This separation is why the same model works for "give me text from a 5 minute clip" and "give me a single 30 s decoding result with full control."

The Transformer model

Defined in whisper/model.py.

graph TD
    mel[mel: B x n_mels x 3000] --> conv1[Conv1d 3, pad 1] --> gelu1[GELU] --> conv2[Conv1d 3, stride 2, pad 1] --> gelu2[GELU]
    gelu2 --> permute[permute B x T x C] --> sine[+ sinusoidal pos. embedding]
    sine --> encblocks[N x ResidualAttentionBlock<br/>self-attn only]
    encblocks --> ln1[LayerNorm] --> features[audio features<br/>B x n_audio_ctx x n_audio_state]

    tokens[tokens: B x L] --> emb[Embedding] --> pos[+ learned pos. embedding]
    pos --> decblocks[N x ResidualAttentionBlock<br/>self-attn + cross-attn]
    features -.->|cross-attn keys/values| decblocks
    decblocks --> ln2[LayerNorm] --> logits[logits = x @ E^T]

Notable details:

• The encoder uses sinusoidal positional embeddings registered as a buffer; the decoder uses learned positional parameters.
• All LayerNorm, Linear, and Conv1d layers cast weights to the activation dtype, which lets the same module run in fp16 or fp32 without rewriting forward methods.
• MultiHeadAttention opportunistically uses PyTorch's scaled_dot_product_attention (since release v20240930); it can be disabled with the disable_sdpa() context manager (used during DTW alignment, where attention weights need to be inspected).
• A KV cache is installed on demand via Whisper.install_kv_cache_hooks(), which registers forward hooks on each attention layer's key and value projections. The hooks save (or concatenate to) cached tensors so that successive autoregressive steps only run the new token through the network. Cleanup unregisters the hooks.
• Whisper.set_alignment_heads() decompresses a base85-gzipped boolean mask of (n_text_layer, n_text_head) and stores it as a sparse buffer. Word timestamps use only these heads. Per-model masks live in _ALIGNMENT_HEADS in whisper/__init__.py.

Audio frontend

whisper/audio.py defines the fixed audio hyperparameters that the rest of the codebase relies on:

Constant	Value	Meaning
SAMPLE_RATE	16000	Hz
N_FFT	400	STFT window size (25 ms)
HOP_LENGTH	160	STFT hop (10 ms)
CHUNK_LENGTH	30	Seconds per inference chunk
N_SAMPLES	480000	CHUNK_LENGTH * SAMPLE_RATE
N_FRAMES	3000	N_SAMPLES / HOP_LENGTH mel frames
FRAMES_PER_SECOND	100	Mel frames per second
TOKENS_PER_SECOND	50	Output tokens per second (50 = 1/0.02 s)

load_audio() shells out to ffmpeg to decode and resample to mono 16 kHz PCM. log_mel_spectrogram() runs PyTorch's torch.stft and applies a precomputed mel filterbank from whisper/assets/mel_filters.npz (which contains both 80- and 128-channel filterbanks; large-v3/turbo use 128).

Tokenizer and special tokens

The tokenizer is a tiktoken.Encoding wrapped by Tokenizer. It loads byte-pair-encoding ranks from whisper/assets/gpt2.tiktoken (English-only) or whisper/assets/multilingual.tiktoken (multilingual). On top of the base vocabulary it adds:

• <|startoftranscript|>, <|endoftext|>, <|startoflm|>, <|startofprev|>, <|nospeech|>, <|notimestamps|>
• One language token per supported language (<|en|>, <|zh|>, …, up to num_languages)
• Task tokens <|transcribe|> and <|translate|>
• 1501 timestamp tokens <|0.00|> … <|30.00|> at 0.02 s resolution

The decoder is conditioned by inserting these tokens into the SOT (start-of-transcript) prefix.

Two compute backends for word timing

Word timestamps come from running DTW on the cross-attention weights of the model's "alignment heads," with a median filter applied along the time axis. The DTW and median filter each have a CPU implementation (Numba JIT in whisper/timing.py) and a Triton CUDA kernel (whisper/triton_ops.py). The CUDA path is tried first and falls back to CPU on RuntimeError/CalledProcessError (e.g. when the Triton toolkit is not present). See Word timestamps.

CLI shape

whisper.transcribe:cli is registered as the whisper console script in pyproject.toml. It parses arguments with argparse, builds a temperature schedule for fallback, loads a model with whisper.load_model(), and then iterates over input audio files, calling transcribe() and a ResultWriter for each. The python -m whisper form is supported via whisper/__main__.py.

What is not here

• No training, fine-tuning, or dataset loading code. The repo only does inference.
• No real-time/streaming pipeline. Audio is processed in 30 s chunks; lower-latency inference would require external tooling.
• No web server, GUI, or REST/RPC layer.
• No model conversion utilities (e.g. ONNX, GGML). The checkpoint format is a plain torch.save dict with dims and model_state_dict.