Normalizers

Active contributors: Jong Wook Kim

Purpose

Normalize transcribed text for evaluation against reference labels — removing punctuation, lowercasing, expanding contractions, converting spelled-out numbers and currencies into digits, and harmonizing British/American spelling. These normalizers are not invoked during normal transcription; they exist so that experiments matching the paper (computing WER against reference transcripts) can use the exact normalization the authors used.

Directory layout

whisper/
└── normalizers/
    ├── __init__.py             # re-exports BasicTextNormalizer, EnglishTextNormalizer
    ├── basic.py                # ~80 lines — Unicode-aware basic normalizer
    ├── english.py              # ~550 lines — English text normalizer (numbers, contractions, abbreviations)
    └── english.json            # spelling dictionary (British -> American, etc.), ~56 KB

Key abstractions

Symbol	File	Description
remove_symbols_and_diacritics(s, keep="")	whisper/normalizers/basic.py	Replaces marks/symbols/punctuation with spaces and strips diacritics. Includes a hand-coded ADDITIONAL_DIACRITICS table for letters NFKD does not decompose (œ→oe, ß→ss, ł→l, etc.).
remove_symbols(s)	whisper/normalizers/basic.py	Same idea but keeps diacritics; uses NFKC instead of NFKD.
BasicTextNormalizer(remove_diacritics=False, split_letters=False)	whisper/normalizers/basic.py	Lowercase → strip bracketed/parenthesized tags → clean (with or without diacritics) → optional letter-split → collapse whitespace. Used for non-English evaluation.
EnglishNumberNormalizer	whisper/normalizers/english.py	Walks token streams to convert spelled-out numbers ("two hundred and three" → "203"), with currency, percent, ordinals, plurals, and digit-spelling cases ("double oh seven" → "007").
EnglishSpellingNormalizer	whisper/normalizers/english.py	Looks up tokens in english.json to apply spelling normalization (e.g. "mobilisation" → "mobilization").
EnglishTextNormalizer	whisper/normalizers/english.py	Top-level English normalizer: lower → strip brackets → expand contractions and abbreviations → run EnglishNumberNormalizer → run EnglishSpellingNormalizer → strip remaining symbols → collapse whitespace.

How it works

The English pipeline is a chain:

graph LR
    A[input text] --> B[lowercase + strip bracketed/parenthesized tags]
    B --> C[contractions: 's, 've, 'll, 'd, ...]
    C --> D[abbreviations: Mr. -> mister, etc.]
    D --> E[EnglishNumberNormalizer<br/>spelled-out numbers, currencies, percents]
    E --> F[EnglishSpellingNormalizer<br/>english.json lookup]
    F --> G[remove_symbols & lowercase]
    G --> H[collapse whitespace]
    H --> I[normalized text]

EnglishNumberNormalizer is the bulk of the file. It maintains hand-built tables of:

• ones / tens / multipliers / ordinals / ones_plural / ones_suffixed for word-to-number mapping.
• Currency symbols and their three-letter codes.

It walks the input as space-separated tokens with a sliding window (more_itertools.windowed) to handle multi-word constructs like "three and a half million" or "twenty-fifth".

Test cases in tests/test_normalizer.py document the supported transformations: "two double o eight" → "2008", "$20 million" → "$20000000", "double oh seven" → "007", "one triple oh one" → "10001", "two dollars and seventy cents" → "$2.70", etc.

BasicTextNormalizer's __call__ does:

s = s.lower()
s = re.sub(r"[<\[][^>\]]*[>\]]", "", s)   # remove <speaker> or [noise]
s = re.sub(r"\(([^)]+?)\)", "", s)         # remove (parenthesized)
s = self.clean(s).lower()
if self.split_letters:
    s = " ".join(regex.findall(r"\X", s, regex.U))   # split on grapheme clusters
s = re.sub(r"\s+", " ", s)                 # collapse whitespace

split_letters=True is intended for character-level WER evaluation in scripts that don't separate words with spaces (e.g. Chinese, Japanese — paired with paper Appendix D.2).

Integration points

• Imports from: re, regex, unicodedata, more_itertools, and whisper/normalizers/english.json.
• Imported by: nothing inside the inference pipeline. Reachable as from whisper.normalizers import EnglishTextNormalizer, BasicTextNormalizer for users running benchmarks.
• Side effects: none.

Entry points for modification

• New language normalizer: add a sibling module to whisper/normalizers/ and re-export from __init__.py.
• Spelling dictionary: edit english.json (UTF-8 JSON of {spelling_a: spelling_b, ...}).
• New currency or number pattern: extend the relevant table in EnglishNumberNormalizer.__init__. Add a corresponding parametric assertion in tests/test_normalizer.py.
• These modules are intentionally not on the inference path; if you need transcript post-processing in whisper.transcribe(), do it in whisper/utils.py instead.

See also: Tokenizer for how raw text is produced from tokens.