Transcribe

Active contributors: Jong Wook Kim, ryanheise, fatih

Purpose

Drive end-to-end transcription of an arbitrary-length audio file. Build the mel spectrogram (with 30 s of zero padding so the loop's last partial chunk is safe), iterate over 30 s windows, call model.decode() with temperature fallback, slice each window's output by predicted timestamp tokens, optionally compute word timestamps, and aggregate everything into {text, segments, language}. Also exposes the whisper CLI.

Directory layout

whisper/
├── __main__.py    # `python -m whisper` -> calls cli()
└── transcribe.py  # transcribe() and cli()

Key abstractions

Symbol	Description
transcribe(model, audio, ...)	The main public function. Bound onto Whisper.transcribe.
decode_with_fallback(segment)	Inner closure: try increasing temperatures until thresholds are met.
new_segment(start, end, tokens, result)	Inner factory for per-segment dicts.
word_anomaly_score(word) / is_segment_anomaly(segment) / next_words_segment(segments)	Heuristics for the hallucination-silence skip path.
cli()	Argparse-based CLI. Registered as the whisper console script.

How it works

graph TD
    A[audio] --> B[log_mel_spectrogram with padding=N_SAMPLES]
    B --> C{language is None?}
    C -- yes --> D[detect on first 30 s] --> E
    C -- no --> E[content_frames = mel.shape - N_FRAMES]
    E --> F[seek_clips from clip_timestamps]
    F --> G{seek loop}
    G --> H[mel_segment = pad_or_trim slice]
    H --> I[decode_with_fallback]
    I --> J{no_speech check}
    J -- skip --> G
    J -- keep --> K[slice tokens at consecutive timestamps]
    K --> L{word_timestamps?}
    L -- yes --> M[add_word_timestamps] --> N[hallucination silence skip]
    L -- no --> O
    N --> O[append segments to all_segments]
    O --> G
    G -- done --> P[return text, segments, language]

Outer loop bookkeeping

The loop is "obscurely flattened" — there is a single while clip_idx < len(seek_clips) that advances both clip_idx and seek (a comment in the code admits this is awkward and a candidate for refactor). Per iteration:

• Decide the current [seek_clip_start, seek_clip_end] range (in mel frames).
• Cut a mel_segment of up to N_FRAMES from mel[:, seek:seek+segment_size], then pad_or_trim to exactly N_FRAMES.
• Build the prompt (either the trailing tokens since the last reset, or the initial-prompt-prefixed prompt when carry_initial_prompt=True).
• Call decode_with_fallback.
• Apply the no_speech_threshold test — if result.no_speech_prob > no_speech_threshold and (when set) avg_logprob is also below logprob_threshold, skip this chunk by advancing seek by segment_size and continue.
• Otherwise, slice the predicted tokens at every pair of consecutive timestamp tokens, building one segment per slice. The segment time range is time_offset + (start_timestamp_pos / TOKENS_PER_SECOND) to ... end_timestamp_pos .... If the chunk ends with a single timestamp (no closing one), advance seek by the full segment_size; otherwise advance to the last successful timestamp position so the next chunk starts cleanly.
• Optionally call add_word_timestamps and run hallucination-silence-skip heuristics.
• Extend all_segments and all_tokens. If condition_on_previous_text=False or temperature exceeded 0.5, reset prompt_reset_since so the next chunk's prompt does not carry forward.

decode_with_fallback

For each t in the temperature schedule (default (0.0, 0.2, 0.4, 0.6, 0.8, 1.0)):

1. Build a DecodingOptions from decode_options with temperature=t. At t > 0, drop beam_size/patience (sampling cannot use beam search). At t == 0, drop best_of.
2. Call model.decode(mel_segment, options).
3. Apply the fallback rules:
   • compression_ratio > compression_ratio_threshold (default 2.4) → too repetitive, retry.
   • avg_logprob < logprob_threshold (default -1.0) → too uncertain, retry.
   • But if no_speech_prob > no_speech_threshold (default 0.6) AND avg_logprob < logprob_threshold → treat the segment as silence; do not retry (the no-speech check at the outer loop will skip it anyway).
4. Return as soon as a temperature passes.

Word-timestamp + hallucination silence

When word_timestamps=True, whisper/timing.py:add_word_timestamps augments each segment with a words list. After that, transcribe():

• Updates seek to the last word's end (when the chunk did not end with a "single timestamp" pattern).
• If hallucination_silence_threshold is set, runs three heuristics:
  1. If the gap between the chunk start and the first word exceeds the threshold AND the first segment scores as anomalous, jump seek past that gap and retry the whole chunk.
  2. For each segment that is itself anomalous and is bracketed by silence (longer than threshold) on both sides, drop it and the segments after it, and seek to its start.
  3. Otherwise, advance seek to the last word's end if it falls inside the window.

is_segment_anomaly looks at the first 8 non-punctuation words. word_anomaly_score(word) adds 1 if its probability < 0.15, plus penalties for very short (< 0.133 s) or very long (> 2.0 s) durations. A segment is anomalous if the cumulative score reaches 3, or matches roughly the count of words.

Empty-segment cleanup

After all the heuristics, any segment with start == end or empty text has its text, tokens, and words zeroed out. They remain in the output (so the indexing stays stable) but contribute nothing to the final transcript text.

Output shape

{
    "text": str,                # concatenation of all non-suppressed tokens decoded
    "segments": [                # list of dicts, each with:
        {
            "id": int,
            "seek": int,         # frame offset where the window started
            "start": float, "end": float,
            "text": str,
            "tokens": List[int],
            "temperature": float,
            "avg_logprob": float,
            "compression_ratio": float,
            "no_speech_prob": float,
            "words": [           # only when word_timestamps=True
                {"word": str, "start": float, "end": float, "probability": float},
                ...
            ],
        },
        ...
    ],
    "language": str,             # ISO code
}

CLI

cli() builds an argparse.ArgumentParser covering:

• audio (positional, one or more files)
• --model (default turbo)
• --model_dir, --device, --output_dir, --output_format, --verbose
• --task, --language
• --temperature, --best_of, --beam_size, --patience, --length_penalty
• --suppress_tokens, --initial_prompt, --carry_initial_prompt
• --condition_on_previous_text, --fp16
• --temperature_increment_on_fallback, --compression_ratio_threshold, --logprob_threshold, --no_speech_threshold
• Word-timestamp flags: --word_timestamps, --prepend_punctuations, --append_punctuations, --highlight_words, --max_line_width, --max_line_count, --max_words_per_line, --hallucination_silence_threshold
• --threads, --clip_timestamps

cli() resolves the temperature schedule (single value vs (t, t+inc, …, 1.0)), loads the model, builds a ResultWriter via whisper/utils.py:get_writer, and iterates over audio_paths. Each file's transcribe() call is wrapped in a try/except that prints the traceback and continues — one bad file should not abort a batch.

Integration points

• Imports from: whisper/audio.py (log_mel_spectrogram, pad_or_trim, N_FRAMES, N_SAMPLES, HOP_LENGTH, SAMPLE_RATE, FRAMES_PER_SECOND); whisper/decoding.py (DecodingOptions, DecodingResult); whisper/timing.py:add_word_timestamps; whisper/tokenizer.py (LANGUAGES, TO_LANGUAGE_CODE, get_tokenizer); whisper/utils.py (exact_div, format_timestamp, get_end, get_writer, make_safe, optional_float, optional_int, str2bool).
• Imported by: whisper/__main__.py (CLI entry); whisper/__init__.py (transcribe is re-exported and bound onto Whisper.transcribe).

Entry points for modification

• New CLI flags: add to cli()'s argparse and thread through to transcribe() (most flags are forwarded as **args).
• Different fallback rules: edit decode_with_fallback. Note the asymmetry between "too repetitive" / "too uncertain" / "silent."
• Different prompt strategy: change how decode_options["prompt"] is built before decode_with_fallback (search for carry_initial_prompt).
• Different segmentation: the slicing happens around consecutive = torch.where(timestamp_tokens[:-1] & timestamp_tokens[1:]). Replacing this is the gateway to alternative segmentation algorithms (e.g. VAD-driven).

See also: Decoding for what model.decode() actually does, and Timing for word-level alignment and DTW.