rust-lang/rust

Frontend: parse, expand, resolve, lower

The "frontend" is everything that turns text into HIR: lex, parse, expand macros, resolve names, and lower the AST. Each step is a separate crate (or two), and each has a well-defined output that the next step consumes.

Pipeline

graph LR
    Bytes["source bytes"] --> Lexer[rustc_lexer]
    Lexer --> Parser[rustc_parse]
    Parser --> AST[(AST)]
    AST --> Validate[rustc_ast_passes]
    Validate --> Expand[rustc_expand]
    Expand -.invokes.-> Builtins[rustc_builtin_macros]
    Expand -.invokes.-> ProcMac[rustc_proc_macro]
    Expand --> Resolve[rustc_resolve]
    Resolve --> Lower[rustc_ast_lowering]
    Lower --> HIR[(HIR)]

Lexer

compiler/rustc_lexer/ is intentionally minimal — it produces a flat stream of tokens with no whitespace or comments dropped (those are kept for diagnostics and rustfmt). It's published as a separate crate on crates.io (rustc-lexer) so that rust-analyzer and other tools can use the same tokenization that rustc does.

It takes raw &str and emits Tokens. Higher-level wrapping (interning into Symbol, attaching spans) happens in rustc_parse.

Parser

compiler/rustc_parse/ implements the full Rust grammar. Output: an ast::Crate. Highlights:

Recovery — the parser tries hard to keep going after a syntax error so users get diagnostics for the rest of the file
Spanned everything — every node carries a span pointing back to source
Macro inputs are kept as token streams (TokenStream) until expansion runs

The AST itself lives in compiler/rustc_ast/ and the format!-string mini-grammar in compiler/rustc_parse_format/.

AST validation

compiler/rustc_ast_passes/ runs immediately after parsing. It checks things that are syntactically valid but semantically forbidden — e.g., pub on a struct field within a pub union, ill-formed feature usage, where clauses on the wrong items. These checks are simpler when done on the AST than on the HIR.

Macro expansion

compiler/rustc_expand/ is the macro engine. It walks the AST, finds mac!() invocations, and asks for their expansion:

macro_rules! macros — expanded purely in the engine itself, by pattern-matching token trees
Built-in macros (format_args!, cfg!, derive(Debug), panic!, include_str!, …) — implemented as Rust code in compiler/rustc_builtin_macros/
#[proc_macro] macros — invoked over the proc-macro server via compiler/rustc_proc_macro/, which talks to a sandboxed user-defined process or DLL

Expansion is fixed-point: a macro can expand to code containing more macros, and the engine keeps going until no invocations remain.

The proc_macro crate (library/proc_macro/) is the user-facing API; the bridge to rustc is in rustc_proc_macro.

Name resolution

compiler/rustc_resolve/ maps every path in the AST to a definition. This includes:

Resolving use declarations
Resolving paths in expressions, types, and patterns
Tracking visibility (pub, pub(crate), pub(in some::path))
Building the module tree (which is interleaved with macro expansion: a macro can expand to mods)
Resolving labels and lifetimes (the lifetime resolver lives in rustc_resolve::late::lifetimes)

Output: Resolutions, attached to the AST so AST lowering can read them.

AST lowering

compiler/rustc_ast_lowering/ is the "AST → HIR" step. HIR (compiler/rustc_hir/) is more uniform and decision-friendly than the AST:

All paths are pre-resolved to DefIds/Res
Sugar is desugared (e.g., for loops become loop { match it.next() { … } }, ? becomes branching, if let and while let are lowered)
async fn is lowered to a synchronous function returning a coroutine
Closures keep enough info for borrowck and codegen but lose source-level pretty-print fidelity

Lowering is the last "syntactic" step — everything after it operates on HIR (or further lowered IRs).

Why these are separate crates

Encapsulation — the parser doesn't know about types; the lowerer doesn't know about source bytes; etc.
Compile time — heavy crates like rustc_parse change rarely; isolating them keeps incremental rebuilds fast.
Reusability — rustc_lexer is published to crates.io as rustc-lexer for use in tools.

Key abstractions

Type	Lives in	What it represents
`Token`	`rustc_lexer::Token`	A single token with its kind
`TokenStream`	`rustc_ast::tokenstream`	A token tree fed to / from a macro
`ast::Crate`	`rustc_ast::ast`	A full parsed crate
`ast::Item`	`rustc_ast::ast`	A top-level item
`Span`	`rustc_span::Span`	Source location
`Symbol`	`rustc_span::Symbol`	An interned identifier string
`Ident`	`rustc_span::Ident`	`Symbol + Span + hygiene info`
`hir::Crate`	`rustc_hir`	Lowered crate
`Res`	`rustc_hir::def::Res`	The resolution of a path
`DefId`	`rustc_hir::def_id`	Stable identifier of a definition

Hygiene

Macro hygiene is handled by expansion contexts attached to spans. Every identifier carries a SyntaxContext that records which macro expansion it was introduced from; resolution uses this to decide whether two foo identifiers refer to the same binding. The implementation lives in rustc_span::hygiene and rustc_resolve::macros.

Entry points for modification

Adding a new built-in macro → rustc_builtin_macros, then register in lib.rs
Adding a parser recovery → rustc_parse — find the relevant parse_* method and add a recovery branch
Lowering a new sugar → rustc_ast_lowering (search for an existing example like lower_for_loop)
New name-resolution rule → rustc_resolve
See Diagnostics for how to wire user-facing errors