openai/whisper

Models

Whisper ships seven model sizes, most with English-only variants. All checkpoints are downloaded on demand from openaipublic.azureedge.net, verified by the SHA-256 fragment in their URL, and cached under ~/.cache/whisper. The download logic is in whisper/__init__.py:_download.

Available models

whisper.available_models() returns these names:

Name	Parameters	English-only?	Multilingual?	n_mels	Approx. VRAM	Relative speed	Notes
`tiny.en`	39 M	✓		80	~1 GB	~10x
`tiny`	39 M		✓	80	~1 GB	~10x
`base.en`	74 M	✓		80	~1 GB	~7x
`base`	74 M		✓	80	~1 GB	~7x
`small.en`	244 M	✓		80	~2 GB	~4x
`small`	244 M		✓	80	~2 GB	~4x
`medium.en`	769 M	✓		80	~5 GB	~2x
`medium`	769 M		✓	80	~5 GB	~2x
`large-v1`	1550 M		✓	80	~10 GB	1x
`large-v2`	1550 M		✓	80	~10 GB	1x	Released Dec 2022.
`large-v3`	1550 M		✓	128	~10 GB	1x	Released Nov 2023.
`large` (alias)	1550 M		✓	128	~10 GB	1x	Currently aliased to `large-v3`.
`large-v3-turbo`	809 M		✓	128	~6 GB	~8x	Distilled `large-v3`. Sep 2024.
`turbo` (alias)	809 M		✓	128	~6 GB	~8x	Default CLI model. No translation support.

The full URL → SHA-256 mapping lives in _MODELS in whisper/__init__.py. Aliases like large and turbo are real keys with the same URL/SHA as their target.

English-only vs multilingual

English-only models (*.en) are trained on English audio, use the GPT-2 BPE tokenizer (whisper/assets/gpt2.tiktoken), and do not have language tokens. They are not capable of language detection or translation.
Multilingual models use a 99-language vocabulary (whisper/assets/multilingual.tiktoken) with one special token per language plus the <|translate|>/<|transcribe|> task tokens.
Whisper.is_multilingual checks n_vocab >= 51865.

The README notes that the English-only tiny.en and base.en models have a meaningful quality advantage over their multilingual counterparts; for small.en and medium.en the difference shrinks.

Translation

All multilingual models can translate to English with --task translate.
The English-only models cannot translate.
The turbo model is not trained for translation — it returns the original language even when --task translate is used. The model card recommends medium or large for translation.

Model dimensions

Every checkpoint stores its ModelDimensions (defined in whisper/model.py) along with the state dict. The dimensions, as observed in the published models:

Field	tiny	base	small	medium	large-v1/v2	large-v3	turbo
`n_mels`	80	80	80	80	80	128	128
`n_audio_ctx`	1500	1500	1500	1500	1500	1500	1500
`n_audio_state`	384	512	768	1024	1280	1280	1280
`n_audio_head`	6	8	12	16	20	20	20
`n_audio_layer`	4	6	12	24	32	32	32
`n_text_ctx`	448	448	448	448	448	448	448
`n_text_state`	384	512	768	1024	1280	1280	1280
`n_text_head`	6	8	12	16	20	20	20
`n_text_layer`	4	6	12	24	32	32	4

The defining property of turbo is its drastically reduced decoder (n_text_layer = 4) — that's where the speed comes from.

Loading a custom checkpoint

whisper.load_model(path) accepts a filesystem path to a .pt produced by:

torch.save({
    "dims": asdict(model.dims),
    "model_state_dict": model.state_dict(),
}, "my_model.pt")

When loaded by path, _ALIGNMENT_HEADS is not applied (it is a per-name lookup). The model's default — last half of decoder layers — is used as the alignment-head mask, so word timestamps will work but with somewhat degraded accuracy. To restore curated heads on a fine-tuned checkpoint, call model.set_alignment_heads(your_dump_bytes) after loading. See Whisper.set_alignment_heads in whisper/model.py.

Model card

For limitations, training data, and intended use, see model-card.md.