Audio processing

Active contributors: Jong Wook Kim

Purpose

Convert any input file (or NumPy/PyTorch array) into the fixed-size log-Mel spectrogram tensor that the model encoder expects: 16 kHz mono PCM → STFT → mel filterbank → log → range-clamped. All hardcoded audio constants (sample rate, FFT size, hop length, chunk length) live here, and other modules import them rather than redefining them.

Directory layout

whisper/
├── audio.py                    # everything in this section
└── assets/
    └── mel_filters.npz         # precomputed 80-mel and 128-mel filterbanks

Key abstractions

Symbol	File / location	Description
SAMPLE_RATE, N_FFT, HOP_LENGTH, CHUNK_LENGTH, N_SAMPLES, N_FRAMES, N_SAMPLES_PER_TOKEN, FRAMES_PER_SECOND, TOKENS_PER_SECOND	whisper/audio.py	Hardcoded audio hyperparameters used across the package. Other modules import them rather than redefining.
load_audio(file, sr=16000)	whisper/audio.py	Decodes any media file via ffmpeg to mono float32 PCM in [-1, 1].
pad_or_trim(array, length=N_SAMPLES, axis=-1)	whisper/audio.py	Right-pads with zeros or truncates to exactly 30 s. Works on both NumPy arrays and PyTorch tensors.
mel_filters(device, n_mels)	whisper/audio.py	LRU-cached loader for the precomputed mel filterbank (whisper/assets/mel_filters.npz).
log_mel_spectrogram(audio, n_mels=80, padding=0, device=None)	whisper/audio.py	Computes a (n_mels, n_frames) log-mel spectrogram from an audio file path, NumPy array, or tensor.

How it works

graph LR
    A[file path / ndarray / tensor] --> B{path?}
    B -- yes --> C[ffmpeg subprocess<br/>-f s16le -ac 1 -ar 16000]
    C --> D[int16 stream] --> E[np.frombuffer<br/>/32768 -> float32]
    B -- no --> E
    E --> F[torch.from_numpy / .to&#40;device&#41;]
    F --> G{padding > 0?}
    G -- yes --> H[F.pad with zeros]
    G -- no --> I
    H --> I[torch.stft<br/>N_FFT=400, HOP_LENGTH=160<br/>hann window]
    I --> J[take complex bins :-1<br/>magnitude squared]
    J --> K[mel_filters @ magnitudes]
    K --> L[clamp >= 1e-10<br/>log10]
    L --> M[clamp >= max - 8.0]
    M --> N[(log_spec + 4.0) / 4.0]
    N --> O[returns shape n_mels x n_frames]

The most common call shape is from transcribe():

mel = log_mel_spectrogram(audio, model.dims.n_mels, padding=N_SAMPLES)

That extra 30 s of zero padding is intentional: it lets the loop slice the final partial chunk without going out of bounds.

load_audio

Builds an ffmpeg argv that:

• Reads any container/codec ffmpeg knows.
• Down-mixes to mono (-ac 1).
• Resamples to sr Hz (-ar).
• Outputs raw signed-16-bit little-endian PCM to stdout (-f s16le -acodec pcm_s16le).

The bytes are then read with np.frombuffer(out, np.int16) and divided by 32768 to land in [-1, 1]. Failures from ffmpeg become a RuntimeError("Failed to load audio: <stderr>"). The function never imports ffmpeg-python — it just shells out, which is why the only system dependency is the ffmpeg binary.

log_mel_spectrogram math

The exact normalization steps after the mel projection are:

log_spec = torch.clamp(mel_spec, min=1e-10).log10()
log_spec = torch.maximum(log_spec, log_spec.max() - 8.0)
log_spec = (log_spec + 4.0) / 4.0

This caps the dynamic range at 80 dB below the per-clip maximum, then linearly maps the resulting interval to roughly [-1, 1]. The result matches what the model was trained on; do not change these constants unless you are also retraining.

Mel filterbanks

whisper/assets/mel_filters.npz is a small file (~4 KB) with two numpy arrays — mel_80 and mel_128 — each precomputed via:

np.savez_compressed(
    "mel_filters.npz",
    mel_80=librosa.filters.mel(sr=16000, n_fft=400, n_mels=80),
    mel_128=librosa.filters.mel(sr=16000, n_fft=400, n_mels=128),
)

so that librosa need not be a runtime dependency. mel_filters() is @lru_cache(maxsize=None)d on (device, n_mels), so each filterbank is loaded and moved to a given device exactly once per process.

Integration points

• Imports from: whisper/utils.py:exact_div (used to derive N_FRAMES, FRAMES_PER_SECOND, TOKENS_PER_SECOND with assertions that the divisions are exact).
• Imported by: whisper/__init__.py (re-exports load_audio, log_mel_spectrogram, pad_or_trim); whisper/transcribe.py (audio frontend for transcribe()); whisper/timing.py (HOP_LENGTH, SAMPLE_RATE, TOKENS_PER_SECOND).
• Side effects: launches an ffmpeg subprocess on path-string input.

Entry points for modification

• New input formats: ffmpeg already handles essentially everything; if a format doesn't work, the fix is on ffmpeg's side, not here.
• Different sample rate: SAMPLE_RATE is referenced everywhere (timing precision, frames/token math). Don't change it without retraining.
• Custom mel banks: drop a new key into assets/mel_filters.npz and add the corresponding n_mels to the assert list in mel_filters().
• Streaming / partial decoding: this module is batch-only; a streaming pipeline would need a new entry point that incrementally extends a buffer and recomputes the STFT on the tail.