Scanner

The scanner (also called the lexer) turns UTF-16 source text into a stream of SyntaxKind tokens. It's the first stage of the compilation pipeline and is shared by parsing, the language service classifier, and JSDoc parsing.

Purpose

Tokens are the unit of work for the parser. The scanner reads characters one at a time, classifies them into tokens (Identifier, OpenBraceToken, NumericLiteral, StringLiteral, JsxText, …), tracks the start and end position of each, and keeps an errorCallback that the parser uses to emit lexical-error diagnostics (e.g., "Unterminated string literal").

The scanner is mode-aware. The same Scanner instance can scan TypeScript, JSX text, regular expressions, and JSDoc comments — switching modes is just a flag toggle.

Source

Single file: src/compiler/scanner.ts (~4,100 lines).

Key abstractions

Symbol	Role
SyntaxKind (in src/compiler/types.ts)	Token enum — punctuation, literals, keywords, contextual keywords
TokenFlags	Bit flags attached to tokens (PrecedingLineBreak, Unterminated, IsRadix, BinarySpecifier, …)
LanguageVariant	Standard or JSX — toggled when entering JSX content
ScriptTarget	ES3 … ESNext — controls keyword sets and identifier rules
Scanner	The stateful object — scan(), lookAhead, tryScan, setText, setLanguageVariant, …
createScanner(languageVersion, skipTrivia, languageVariant?, textInitial?, onError?, start?, length?)	Factory
tokenIsIdentifierOrKeyword(kind)	Common predicate used by the parser
getSpellingSuggestion(name, candidates, getName)	Levenshtein-based hints surfaced as "Did you mean X?" diagnostics

TokenFlags is the place where lexical anomalies are recorded so the parser doesn't need to re-walk the text. For instance, the parser knows whether two tokens have a line break between them by checking TokenFlags.PrecedingLineBreak, which the scanner sets whenever it skips over a newline.

How it works

graph LR
    Source["UTF-16 source text"] --> Scanner["Scanner.scan()"]
    Scanner --> Token["SyntaxKind + value + flags + range"]
    Scanner -. mode flag .-> JSX["JSX scanner subroutine"]
    Scanner -. mode flag .-> RE["regex scanner subroutine"]
    Scanner -. JSDoc parser pulls .-> JSDoc["JSDoc scanner subroutine"]

Internally the scanner is a tight state machine over string indices. The hot loop is scan(). Identifiers are scanned with code-point arithmetic and a precomputed Unicode identifier-part table; surrogate pairs are handled inline. Keyword recognition is a single Map<string, SyntaxKind> lookup keyed on the identifier text.

The scanner is not incremental at the character level — every change re-scans the file. Incrementality lives in the parser, which reuses unchanged subtrees (and their pre-scanned tokens are implicit in the AST node text ranges).

Trivia

"Trivia" is the scanner's term for whitespace, comments, and other non-token characters. By default the scanner skips trivia and reports only meaningful tokens; with skipTrivia=false (used by the formatter and a few code fixes) it surfaces SingleLineCommentTrivia, MultiLineCommentTrivia, NewLineTrivia, WhitespaceTrivia, ShebangTrivia, and ConflictMarkerTrivia. The conflict-marker token exists so editors can keep providing IntelliSense in files with unmerged Git markers.

Special tokens

A few enum members exist only because of edge cases:

• NonTextFileMarkerTrivia — the parser may receive a binary file by mistake; the scanner emits this kind to bail out cleanly.
• BacktickToken, HashToken — only the JSDoc scanner produces these. The regular scanner produces NoSubstitutionTemplateLiteral and PrivateIdentifier instead.
• JsxText, JsxTextAllWhiteSpaces — emitted while scanning the contents of a JSX element.

Integration points

• Parser (src/compiler/parser.ts) creates a Scanner per source file and pulls tokens via nextToken() / lookAhead().
• Classifier in the language service (src/services/classifier.ts and classifier2020.ts) drives semantic and syntactic colourisation by re-scanning files with skipTrivia=false.
• Formatter (src/services/formatting/) walks tokens (with trivia) to compute indentation and spacing.
• JSDoc parser is a sub-mode of the same scanner, switched on inside parser.ts when entering a /** … */ comment.
• Pretty-printer / emitter uses scanner helpers to decide where comments belong on output.

Entry points for modification

Adding a new keyword or token kind:

1. Add the entry to SyntaxKind in src/compiler/types.ts, keeping the keyword block contiguous (token ordering matters — tokenIsIdentifierOrKeyword relies on it).
2. Wire it into the keyword Map and scan() switch in src/compiler/scanner.ts.
3. Update the parser to consume it in the right grammar productions.
4. Update src/compiler/factory/nodeFactory.ts and nodeTests.ts if the new token implies a new node type.
5. Update the emitter and any transformers that need to print or rewrite the new construct.

See primitives/node for the broader story of how scanner output becomes typed nodes.