Transcription

The headline capability: take an audio file in any format ffmpeg can decode, in any of 99 supported languages, and produce a text transcript with segment-level timestamps. Implemented end-to-end by whisper/transcribe.py:transcribe(), which is bound onto Whisper.transcribe.

Surfaces

CLI:

whisper audio.mp3 --model turbo
whisper audio.mp3 --model medium --language Japanese
whisper audio.mp3 --model turbo --output_format srt

Python:

import whisper
model = whisper.load_model("turbo")
result = model.transcribe("audio.mp3")
print(result["text"])
for s in result["segments"]:
    print(s["start"], s["end"], s["text"])

End-to-end flow

graph TD
    A[audio file] --> B[load_audio<br/>ffmpeg -> 16 kHz mono PCM]
    B --> C[log_mel_spectrogram<br/>STFT + 80/128 mel + log]
    C --> D{language given?}
    D -- no --> E[detect_language<br/>on first 30 s]
    D -- yes --> F
    E --> F[seek_clips from --clip_timestamps]
    F --> G{loop over 30 s windows}
    G --> H[decode_with_fallback<br/>temperatures 0.0 -> 1.0]
    H --> I[no_speech check<br/>maybe skip]
    I --> J[slice on consecutive timestamp tokens<br/>build segments]
    J --> K[optional add_word_timestamps]
    K --> L[append to all_segments<br/>advance seek]
    L --> G
    G -- done --> M[result dict<br/>text + segments + language]
    M --> N[ResultWriter<br/>txt/vtt/srt/tsv/json]

Knobs

The most useful options users actually reach for:

Option	Default	Effect
--model	turbo	Model size. See Reference: models.
--language	auto-detect	Skip language detection.
--task	transcribe	Switch to translate for X→English. Use a non-turbo multilingual model for this.
--output_format	all	Pick one of txt/vtt/srt/tsv/json, or all for everything.
--initial_prompt	None	Bias the first window with arbitrary text (vocabulary, names).
--carry_initial_prompt	False	Prepend the initial prompt to every internal decode call instead of letting it scroll out.
--condition_on_previous_text	True	Pass previous text as context for the next window. Disabling reduces failure-loop stickiness at the cost of cross-window consistency.
--temperature / --temperature_increment_on_fallback	0 / 0.2	Build the fallback temperature schedule.
--compression_ratio_threshold	2.4	Trigger fallback when output is too repetitive.
--logprob_threshold	-1.0	Trigger fallback when output is too uncertain.
--no_speech_threshold	0.6	Skip a chunk as silence when the no-speech token's probability is high enough.
--word_timestamps	False	Compute per-word timestamps. See Word timestamps.
--clip_timestamps	0	Comma-separated start,end,start,end,… to transcribe only specific spans.
--fp16	True	Use fp16 (CPU forces fp32 with a warning).

For the full set, see Reference: configuration.

Sliding window and timestamp slicing

The model produces timestamp tokens (<|0.00|>, <|0.02|>, …, <|30.00|>) interleaved with text. transcribe() slices the per-window output every time it sees two consecutive timestamp tokens — that pair brackets a segment. The "single timestamp ending" case (one timestamp at the end with no closing one) is taken as a signal that no more speech happened in the window, and the seek pointer advances by the full segment size; otherwise the seek advances to the last successful timestamp position so the next window starts cleanly.

Temperature fallback

decode_with_fallback retries with the next temperature when the previous decode produced output that looks broken (high compression ratio = likely repetition; very low log-prob = likely garbage), unless the no-speech probability is high (in which case it's accepted as silence). At t > 0, beam_size/patience are dropped because beam search at non-zero temperature is meaningless; at t == 0, best_of is dropped for the same reason.

This is the main reason Whisper feels robust: most tricky chunks (background music, unusual accents, brief stretches of silence) get retried with progressively more random sampling and one of those retries usually produces sensible output.

Output result

{
    "text": "...",                    # full transcript
    "segments": [
        {
            "id": int, "seek": int,
            "start": float, "end": float,
            "text": str, "tokens": List[int],
            "temperature": float, "avg_logprob": float,
            "compression_ratio": float, "no_speech_prob": float,
            "words": [...],            # only if word_timestamps=True
        },
        ...
    ],
    "language": str,                   # ISO 639-1 code
}

See also

• Decoding — the per-window machinery (DecodingTask, beam search, logit filters).
• Transcribe — the full source-level walk-through of the file-level loop.
• Audio processing — what the input mel actually contains.
• Word timestamps — fine-grained timing on top of segment timing.