Testing

The test suite is small and focused. It exercises each major subsystem end-to-end on a real audio fixture, with a few unit tests for the tokenizer, normalizers, and the DTW / median-filter primitives.

Layout

File	What it covers
tests/conftest.py	random fixture (seeds Python and NumPy RNGs to 42); registers the requires_cuda marker
tests/test_audio.py	load_audio and log_mel_spectrogram produce identical results from path vs ndarray
tests/test_tokenizer.py	English-only vs multilingual tokenizer round-trips; Korean text encodes more compactly with multilingual; unicode-aware split
tests/test_timing.py	DTW backtrace correctness on synthetic costs; CPU vs scipy median-filter equivalence; CUDA equivalence (gated)
tests/test_normalizer.py	English number / spelling / text normalization fixtures
tests/test_transcribe.py	End-to-end model load + transcribe on JFK clip with word timestamps, parameterized over every available model
tests/jfk.flac	11-second JFK inaugural clip used as audio fixture

Running

# CPU-only, full test suite minus the heaviest transcribes
pytest -m 'not requires_cuda'

# What CI runs
pytest --durations=0 -vv \
  -k 'not test_transcribe or test_transcribe[tiny] or test_transcribe[tiny.en]' \
  -m 'not requires_cuda'

# A single file
pytest tests/test_normalizer.py -vv

# A single parameterization
pytest "tests/test_transcribe.py::test_transcribe[tiny.en]"

Markers

• @pytest.mark.requires_cuda — registered in tests/conftest.py so it does not warn under strict mode. Used by test_dtw_cuda_equivalence and test_median_filter_equivalence in tests/test_timing.py. CI excludes these with -m 'not requires_cuda'.

What test_transcribe checks

test_transcribe is the broadest test in the suite. For every model returned by whisper.available_models() it:

1. Loads the model on CUDA if available, else CPU.
2. Calls model.transcribe(jfk.flac, temperature=0.0, word_timestamps=True).
3. Asserts the detected language is English.
4. Asserts known phrases ("my fellow americans", "your country", "do for you") are present in the transcription.
5. Round-trips the segment tokens through the tokenizer and asserts equality with result["text"].
6. Verifies that decode_with_timestamps output starts with <|0.00|>.
7. Walks the words array and confirms start < end for every word, and that the word Americans straddles t=1.8 s.

This test downloads the model checkpoint into the local ~/.cache/whisper. It is slow for large-v* models, which is why CI restricts it to tiny / tiny.en.

Adding tests

When changing behavior:

• Audio frontend (whisper/audio.py): extend tests/test_audio.py. Use the JFK fixture; do not commit new audio files unless absolutely necessary.
• Decoding (whisper/decoding.py): prefer adding a focused test against a small model and a known utterance. Avoid asserting on exact token IDs or text that depend on sampling — assert on properties (e.g. detected language, monotone timestamps).
• Word timestamps (whisper/timing.py): prefer synthetic numerical tests like the existing DTW / median-filter cases, which do not need a model load.
• Tokenizer (whisper/tokenizer.py): round-trip-style tests are most stable.
• Normalizers: keep parametric assertions on small input/output pairs, mirroring the existing patterns.

Determinism

tests/conftest.py defines a random fixture that seeds Python's random and NumPy's RNG to 42. PyTorch RNG is not seeded by the fixture; tests that rely on PyTorch sampling behavior should explicitly seed it themselves. Beam search at temperature=0 is deterministic and is what the existing transcribe test relies on.