Language detection

Whisper can identify the spoken language from the first 30 s of audio. Detection is built on the same encoder/decoder model: the decoder is given just <|startoftranscript|> and the logits for the next token are masked to the set of language tokens. The argmax over that masked distribution is the detected language.

Surfaces

Implicit (transcribe / translate without a --language):

whisper audio.mp3        # auto-detect

result = model.transcribe("audio.mp3")  # detection happens internally
print(result["language"])               # ISO 639-1 code

Explicit (low-level Python API):

import whisper
model = whisper.load_model("turbo")
audio = whisper.load_audio("audio.mp3")
audio = whisper.pad_or_trim(audio)
mel = whisper.log_mel_spectrogram(audio, n_mels=model.dims.n_mels).to(model.device)

token, probs = model.detect_language(mel)
print("Language:", max(probs, key=probs.get))
print("Top 5:", sorted(probs.items(), key=lambda kv: -kv[1])[:5])

probs is a dict from ISO 639-1 code (e.g. "en", "ja", "yue") to probability.

How it works

sequenceDiagram
    autonumber
    participant U as caller
    participant T as detect_language()
    participant E as model.encoder
    participant D as model.decoder
    U->>T: model, mel
    T->>T: get_tokenizer(multilingual, num_languages)
    T->>E: encode mel to audio_features<br/>(unless already encoded)
    T->>D: tokens=[[<|startoftranscript|>]] x N<br/>logits[:, 0]
    D-->>T: logits at SOT
    T->>T: mask non-language tokens to -inf
    T->>T: softmax over remaining<br/>argmax = language token
    T-->>U: language_tokens, language_probs

The implementation is in whisper/decoding.py:detect_language:

mask = torch.ones(logits.shape[-1], dtype=torch.bool)
mask[list(tokenizer.all_language_tokens)] = False
logits[:, mask] = -np.inf
language_tokens = logits.argmax(dim=-1)
language_token_probs = logits.softmax(dim=-1).cpu()

tokenizer.all_language_tokens is a tuple of every <|en|>, <|zh|>, …, <|yue|> ID, derived from LANGUAGES in whisper/tokenizer.py and capped at the model's num_languages.

Multilingual only

detect_language raises ValueError("This model doesn't have language tokens so it can't perform lang id") for English-only models — they don't have language tokens in their vocabulary. Whisper.is_multilingual returns True when n_vocab >= 51865.

Inside transcribe()

When language is omitted, whisper/transcribe.py detects the language once on the first 30 s window, prints "Detected language: <Language>" (when verbose is not False), and pins that language for the rest of the file. There is no per-window relabeling, so a multilingual file (rare) will use whichever language the model picks for the start. If you need multilingual mid-file behavior you would have to call detect_language() manually per chunk.

How task="lang_id" differs

DecodingOptions(task="lang_id") runs the same code path inside DecodingTask.run and short-circuits before the main decoding loop, returning a DecodingResult whose language_probs field is populated and whose tokens/text are empty. Use this if you want the full probability dict via the high-level decode() API.

See also

• Tokenizer — how language tokens are added to the vocabulary.
• Decoding — the broader DecodingTask.run pipeline that calls _detect_language.