ruff_python_parser

A handwritten Python parser. Source: crates/ruff_python_parser/.

Purpose

Lex and parse Python source into a ruff_python_ast::Mod. Designed to be fast, error-recovering, and stable: it does not depend on any external grammar generator.

History

Ruff originally used a parser derived from RustPython. In mid-2023 (first commit July 2023) the parser was reimplemented from scratch, giving the project full control over recovery, AST shape, and performance characteristics. The new parser is what every other crate consumes.

Directory layout

crates/ruff_python_parser/
├── src/
│   ├── lib.rs
│   ├── lexer.rs / lexer/
│   ├── parser/         # parser implementation, one module per construct
│   ├── token.rs
│   ├── error.rs
│   ├── string.rs
│   ├── soft_keywords.rs
│   └── ...
├── resources/          # corpus and fixtures
└── tests/              # integration tests

Key abstractions

Type	Purpose
parse_module(source) -> Result<Parsed<Mod>, ParseError>	Top-level entry point for a complete file.
parse_expression(source)	Entry point for an expression-only context.
Parsed<T>	Wraps the result with token stream and source range.
Token, TokenKind	Lexer output.
ParseError / ParseErrorKind	Recoverable error kinds, with span + message.

Permissive parsing

The parser recovers from common errors so that downstream tools can still report useful diagnostics. Examples:

• A missing colon at the end of a def/if/while header is reported as a ParseError but the parser continues.
• Trailing commas, unclosed brackets, and stray indentation are handled gracefully where possible.

This matters for the LSP: a partially-typed file should still produce useful diagnostics rather than collapsing to a single fatal error.

Token stream

The lexer is implemented in lexer.rs and lexer/. It produces Tokens with TextRange spans. The token stream is exposed alongside the AST in Parsed, because some downstream consumers (formatter, noqa parser, logical-line checkers) need raw tokens, not just the tree.

Soft keywords

Python's match/case/type/type_alias are soft keywords (contextual). soft_keywords.rs handles disambiguation.

Round-trip property

cargo dev round-trip parses, unparses, and reparses files from a curated corpus to verify that the AST round-trips through ruff_python_codegen losslessly. Parser changes that break round-trip will fail this check.

Integration points

• Consumed by everything: linter, formatter, ty, all WASM bindings.
• Tested by crates/ruff_python_ast_integration_tests and crates/ruff_python_trivia_integration_tests.
• Fuzzed via targets in fuzz/.

Performance

The parser is one of the hottest paths in the linter. Key choices:

• Hand-written recursive descent for predictability.
• Allocation-light: relies on Vec for child lists and Box<[T]> for finalized nodes.
• The lexer is single-pass and avoids building intermediate token strings.

Modifying the parser

• New syntax (e.g. when supporting a new Python version): extend parser/ and the lexer, add fixtures under resources/, run cargo dev round-trip.
• New error kinds: add a ParseErrorKind, implement the recovery, add fixture cases.
• Always run cargo nextest run -p ruff_python_parser plus the integration crate.
• After parser changes, run the broader test suite — the linter and formatter snapshots are sensitive to AST shape changes.

See ruff_python_ast for the data the parser produces.