Word timestamps

When word_timestamps=True, Whisper produces a per-word start, end, and probability for every word in every segment. The mechanism is DTW alignment on cross-attention weights, restricted to a curated subset of attention heads ("alignment heads") whose attention pattern is known to track audio↔text alignment.

Surfaces

CLI:

whisper audio.mp3 --word_timestamps True
whisper audio.mp3 --word_timestamps True --highlight_words True   # with VTT/SRT underlines
whisper audio.mp3 --word_timestamps True --max_words_per_line 5

Python:

result = model.transcribe("audio.mp3", word_timestamps=True)
for segment in result["segments"]:
    for w in segment["words"]:
        print(f"{w['start']:.2f}-{w['end']:.2f}  ({w['probability']:.2f})  {w['word']}")

Each segment dict gains a "words" list with entries:

{"word": " hello", "start": 0.42, "end": 0.86, "probability": 0.92}

How it works (one-paragraph version)

For each chunk, after the regular decoding loop, whisper/timing.py:add_word_timestamps reruns the model on the same mel + the predicted text tokens, with disable_sdpa() so the manual attention path returns real QK weights. It collects the cross-attention QKs at the heads named in model.alignment_heads, smooths them with a width-7 median filter, averages across heads, and runs DTW on the resulting (text_tokens × time_frames) matrix. The DTW path tells which time frame corresponds to which token boundary; word timings come from grouping those boundaries by Tokenizer.split_to_word_tokens.

For the full implementation walk-through see Timing.

Alignment heads

Each model has a different "best" subset of decoder cross-attention heads for alignment. The masks live in _ALIGNMENT_HEADS in whisper/__init__.py as base85+gzip-compressed boolean arrays of shape (n_text_layer, n_text_head). Whisper.set_alignment_heads() in whisper/model.py decodes them into a sparse boolean buffer used by find_alignment to pick which heads to look at.

If you load a custom checkpoint by path, no alignment heads are set, and the model defaults to "the last half of decoder layers" (set in Whisper.__init__). DTW will still work but accuracy will likely be worse than with curated heads.

CPU and GPU paths

Both DTW and the median filter have GPU implementations in whisper/triton_ops.py:

• dtw_cuda runs dtw_kernel on a "skewed" cost matrix so each anti-diagonal lives in one row, then runs the (cheap) backtrace on CPU.
• median_filter_cuda runs median_kernel(filter_width), a Triton JIT function whose source is template-substituted with bubble-sort iterations specialized to the requested width.

Both fall back to the CPU implementation on RuntimeError or subprocess.CalledProcessError (most often raised when the Triton/CUDA toolkit is not available). The fallback is silent except for a warnings.warn("Failed to launch Triton kernels, ...; falling back to a slower ... implementation...").

Hallucination silence skipping

When --hallucination_silence_threshold <seconds> is also set, transcribe() runs heuristics on the alignment results to skip silent stretches that are likely to make the model hallucinate text. The heuristic is in whisper/transcribe.py:

• word_anomaly_score(word) adds 1 if probability < 0.15, plus penalties for very short (<0.133 s) or very long (>2.0 s) durations.
• is_segment_anomaly(segment) looks at the first 8 non-punctuation words and considers the segment anomalous if their cumulative score reaches 3 or matches the word count.
• If the first segment in the chunk is anomalous and there is more than threshold seconds of silence before it, the seek pointer jumps past the silence and the chunk is retried.
• For middle segments that are anomalous and bracketed by silence on both sides, the segment and everything after it is dropped, and seek is moved to the segment's start.

CLI rendering options

Several CLI flags are only meaningful with --word_timestamps True:

Flag	Effect
--highlight_words True	Underline the active word in VTT/SRT output by emitting one cue per word.
--max_line_width <int>	Wrap subtitle lines at this many characters.
--max_line_count <int>	Limit subtitle cues to N lines (requires --max_line_width).
--max_words_per_line N	Pack at most N words per cue (no effect with --max_line_width).
--prepend_punctuations	Punctuation symbols to merge with the next word (default "\"'"¿([{-).
--append_punctuations	Punctuation symbols to merge with the previous word (default "\"'.。,，!！?？:：")]}、).

The rendering happens in SubtitlesWriter.iterate_result in whisper/utils.py.

Caveats

• Translation + word timestamps: a warnings.warn("Word-level timestamps on translations may not be reliable.") is emitted in whisper/transcribe.py. The 1:N audio↔text relationship in translation breaks the alignment assumption.
• Custom checkpoints (loaded by path) without explicit alignment heads will use the default mask (last half of decoder layers) — accuracy may degrade.
• Word boundaries are quantized to ~50 frames/s of cross-attention output (= 0.02 s).

See also

• Timing — implementation walk-through, including DTW math.
• Model — set_alignment_heads, disable_sdpa, KV-cache hooks.
• Transcribe — where add_word_timestamps is called and how hallucination silence skipping fits into the loop.