Parser

The CPython parser turns a Python source file (or -c string, or REPL line) into an Abstract Syntax Tree. It is a PEG parser generated from Grammar/python.gram; this replaced the historical LL(1) parser in 3.9 (PEP 617). The authoritative reference is InternalDocs/parser.md.

Pipeline

graph LR
    SRC[Source bytes] --> LEXER[Parser/lexer]
    LEXER --> TOKENS[Token stream]
    TOKENS --> PARSER[Parser/parser.c]
    PARSER --> ACTIONS[Parser/action_helpers.c]
    ACTIONS --> AST[AST node]
    GRAMMAR[Grammar/python.gram] -.->|peg_generator| PARSER
    ASDL[Parser/Python.asdl] -.->|asdl_c.py| ASTC[Python/Python-ast.c]
    ASTC --> AST

Directory layout

Path	Role
Grammar/python.gram	The PEG grammar. Editing this is how you add new syntax.
Grammar/Tokens	Names and string forms for terminal tokens (NAME, NUMBER, :, :=, **, …).
Parser/Python.asdl	The AST schema (Zephyr ASDL).
Parser/parser.c	Generated PEG parser (~39k lines).
Parser/pegen.c / pegen.h	Hand-written runtime support for the generated parser (memoization cache, error recovery).
Parser/pegen_errors.c	Error-message synthesis after a parse failure.
Parser/action_helpers.c	Hand-written helpers invoked from grammar action blocks.
Parser/string_parser.c	Parses string literals (including f"", byte strings, raw, …).
Parser/tokenizer/	The byte-level tokenizer (encoding sniffing, indentation, line continuation).
Parser/lexer/	Token-level lexer that wraps the tokenizer with PEG-friendly hooks.
Tools/peg_generator/	The Python implementation of the PEG generator.
Tools/cases_generator/	Not parser-related, but lives nearby and serves a similar "DSL → C" role.

Tokenizer

Tokenization is a byte-level pass invoked from Parser/tokenizer/. It handles:

• Encoding detection — the # -*- coding: ... -*- cookie or BOM (tokenizer/utf8_helpers.c and friends).
• Indentation tracking — emits INDENT/DEDENT synthetic tokens.
• Continuation lines — \-continuation, parenthesized continuation.
• String prefix recognition — produces a single STRING token; the parser later calls into Parser/string_parser.c to parse f-strings recursively.

The tokenizer is exposed to Python through Python/Python-tokenize.c, which is what tokenize.tokenize ultimately calls.

PEG parser

CPython uses an ordered-choice PEG. Grammar rules look like:

simple_stmt[stmt_ty]:
    | assignment
    | star_expressions
    | return_stmt
    | import_stmt
    | ...

The bracketed [stmt_ty] is the C return type of the rule. Action blocks { ... } after a rule alternative are C expressions that build the AST node:

return_stmt[stmt_ty]:
    | 'return' a=[star_expressions] { _PyAST_Return(a, EXTRA) }

EXTRA expands to lineno, col_offset, end_lineno, end_col_offset, p->arena — the helpers in Parser/action_helpers.c and the macros in Parser/pegen.h take care of the rest.

The parser is recursive-descent with packrat memoization in pegen.c so that the worst-case is linear in input size. Memoization keys are (rule, position) pairs and the cache is per-parse.

When parsing fails, Parser/pegen_errors.c walks the deepest reached position to produce a useful error message — the famous "expected :" pointing at the right column. The newer "did you mean..." suggestions are in Python/suggestions.c.

AST construction

The AST schema is defined in Parser/Python.asdl using ASDL (a small DSL invented by the Zephyr Compiler Group). Each grammar action ultimately calls into a generated _PyAST_* constructor (e.g. _PyAST_FunctionDef, _PyAST_BinOp, _PyAST_Constant).

The AST itself is implemented in Python/Python-ast.c and the Python-visible classes in Lib/ast.py. Both are generated from Python.asdl via Parser/asdl_c.py. Adding a new node means: edit the ASDL file → run make regen-ast → use it in the grammar.

ast.unparse(...) (the inverse — AST → source) is implemented in Lib/_ast_unparse.py; CPython itself rarely needs it but it's used by the ast module and many tools.

How a .py file flows

sequenceDiagram
    participant File
    participant Tokenizer as Parser/tokenizer
    participant Lexer as Parser/lexer
    participant PEG as Parser/parser.c
    participant Helpers as action_helpers.c
    participant AST as Python/Python-ast.c
    File ->> Tokenizer: bytes
    Tokenizer ->> Lexer: tokens
    Lexer ->> PEG: token stream
    PEG ->> Helpers: action callbacks
    Helpers ->> AST: _PyAST_*(...)
    AST -->> PEG: PyObject* (mod_ty)
    PEG -->> File: AST root

The entry points used by the rest of the runtime are:

• _PyParser_ASTFromString(...) — used by compile(), the REPL, and eval().
• _PyParser_ASTFromFile(...) — used when CPython is imported a .py file.
• _PyParser_TokenizerFromString(...) — used by tokenize and IDLE.

Changing the grammar

The full procedure is in InternalDocs/changing_grammar.md. In short:

1. Edit Grammar/python.gram to add the rule.
2. (Optional) Add tokens to Grammar/Tokens and run make regen-token.
3. Add the AST node type to Parser/Python.asdl and run make regen-ast.
4. Run make regen-pegen to regenerate Parser/parser.c.
5. Wire AST → bytecode in Python/codegen.c (see Compiler).
6. Add a test in Lib/test/test_grammar.py, test_compile.py, and a parser test under Lib/test/test_peg_*.py.
7. Update Doc/reference/grammar.rst (the user-facing grammar) and the language reference under Doc/reference/.
8. Add a Misc/NEWS.d/next/Core_and_Builtins/...rst entry.

Entry points for modification

• New syntactic construct → Grammar/python.gram + Parser/Python.asdl + Python/codegen.c.
• Better error message → Parser/pegen_errors.c + Python/suggestions.c.
• Tokenizer change → Parser/tokenizer/, test in test_tokenize.py.
• Argument _PyParser_* API → Parser/peg_api.c.