Compiler

After the parser returns an AST, the compiler turns it into a PyCodeObject — the unit of code that the interpreter actually executes. The pipeline has four stages: AST → instruction sequence → control-flow graph → bytecode. The authoritative reference is InternalDocs/compiler.md.

Pipeline

graph LR
    AST -->|symtable.c| SYM[Symbol table]
    AST -->|codegen.c| INSTRS[Instruction sequence]
    SYM --> INSTRS
    INSTRS -->|flowgraph.c| CFG[Control-flow graph]
    CFG -->|optimizations| CFGO[Optimized CFG]
    CFGO -->|assemble.c| BC[Bytecode + metadata]
    BC --> CODE[PyCodeObject]

Files

File	Responsibility
Python/symtable.c	Walks the AST to produce per-scope symbol tables (used to choose STORE_NAME vs STORE_FAST, decide cell vs free vars, mark globals/nonlocals).
Python/compile.c	The top-level driver: takes an AST, runs the symbol table pass, dispatches to codegen and assembly.
Python/codegen.c	Emits the instruction sequence — a flat list of _PyInstructions — by walking the AST.
Python/flowgraph.c	Builds and optimizes the CFG (peephole, jump threading, dead code, constant folding for tuples and consts).
Python/instruction_sequence.c	The data structure compile.c and codegen.c manipulate before flowgraph.
Python/assemble.c	Converts the optimized CFG into the byte arrays that live on the PyCodeObject.
Python/ast.c	AST validation (e.g. del 1 is rejected here, not in the grammar).
Python/ast_preprocess.c	Transformations on the AST before codegen (constant folding of literals).
Python/future.c	Detects from __future__ import ... and toggles compiler flags.
Objects/codeobject.c	The runtime representation: PyCodeObject, co_consts, co_varnames, the line table, the exception table.

The AST → codeobject path is invoked through _PyAST_Compile() in Python/compile.c, which is what compile() and the import machinery call.

Symbol table pass

Before generating any instructions, the compiler walks the AST to assign each name to a scope: local, cell, free, global, or class-local. This is what tells codegen to emit LOAD_FAST vs LOAD_DEREF vs LOAD_GLOBAL. The pass also detects illegal patterns like nonlocal x outside a function, and computes which functions need __class__ cells (for zero-argument super()).

Implementation: Python/symtable.c. The Python wrapper is symtable.symtable (Lib/symtable.py), which is reached through Modules/symtablemodule.c.

Code generation

Python/codegen.c is structured as a recursive walk over the AST: codegen_visit_stmt(stmt_ty), codegen_visit_expr(expr_ty), etc. Each clause emits one or more instructions into the current _PyInstructionSequence. Examples of patterns you will see:

• Boolean short-circuit: emits JUMP_IF_FALSE_OR_POP / JUMP_IF_TRUE_OR_POP to leave the operand on the stack on the short-circuit path.
• with statements: emit a setup that registers the context manager with the exception table; the corresponding cleanup is patched up by the flowgraph.
• Comprehensions: a lambda-like nested function (<listcomp>, <dictcomp>, …) is compiled and embedded as a constant.
• Pattern matching: match statements produce a small decision tree using the new MATCH_* opcodes.

Because the bytecode has limited oparg width (8 bits), Python/codegen.c emits EXTENDED_ARG prefix instructions for wide operands. The flowgraph and assembler pass over them later if jump distances change.

Control-flow graph and optimization

After codegen, Python/flowgraph.c builds a CFG of basic blocks. Several optimizations run on it:

• Constant folding — constants in tuples, frozen sets, and a few binary ops.
• Dead code elimination — unreachable blocks (after return, raise, infinite loops).
• Jump threading — if a: jump X; X: jump Y becomes if a: jump Y.
• NOP removal — instructions used only to anchor line numbers are kept; pure NOPs aren't.
• Stack effect verification — every block must leave the value stack at a consistent depth; mismatches are caught here.
• Liveness analysis — used to mark variables as dead and emit STORE_FAST_MAYBE_NULL for variables that may not be initialized.

This is a very active area: Python/flowgraph.c is one of the largest files in Python/ (~135k lines including generated tables), and the transformation set has grown substantially since 3.11.

Assembly

Python/assemble.c takes the optimized CFG and produces the byte arrays that live on PyCodeObject:

• co_code — the raw 16-bit code units (opcode << 8 | oparg).
• co_consts — tuple of constants referenced by LOAD_CONST and RESUME.
• co_names — names referenced by LOAD_NAME / LOAD_GLOBAL.
• co_varnames — local variable names (co_nlocals).
• co_cellvars / co_freevars — closure variables.
• co_linetable — compact line-number table (PEP 626).
• co_exceptiontable — the exception handlers (PEP 657-ish, see InternalDocs/exception_handling.md).
• Inline cache slots between specialized instructions — sized from Lib/_opcode_metadata.py.

Code object layout

graph TD
    Code[PyCodeObject] --> Code1[co_code: bytes]
    Code --> Code2[co_consts: tuple]
    Code --> Code3[co_names / co_varnames / co_cellvars / co_freevars]
    Code --> Code4[co_linetable / co_exceptiontable]
    Code --> Code5[co_executors: array of optimizer outputs]
    Code --> Code6[co_qualname / co_filename / co_firstlineno]

co_executors is a per-code array of tier-2 executors that get patched in for hot loops at runtime. The list of opcode names and their inline-cache widths is in Lib/_opcode_metadata.py; both are generated from Python/bytecodes.c.

Adding a new opcode

1. Add the opcode definition to Python/bytecodes.c — this is a small DSL (described in Tools/cases_generator/interpreter_definition.md) that the cases_generator turns into the dispatch table.
2. Run make regen-cases. This regenerates:
   • Python/generated_cases.c.h — the tier-1 dispatch.
   • Python/executor_cases.c.h — the tier-2 (uop) dispatch.
   • Python/optimizer_cases.c.h — the optimizer's abstract interpretation cases.
   • Include/internal/pycore_uop_ids.h and _metadata.h.
   • Lib/_opcode_metadata.py.
3. Emit it from Python/codegen.c.
4. Update Python/specialize.c if you want a specialized variant.
5. Add a test in Lib/test/test_dis.py (the disassembler) and Lib/test/test_capi/test_opt.py (the optimizer).

Entry points for modification

• AST → bytecode behavior — Python/codegen.c.
• Peephole / CFG transforms — Python/flowgraph.c.
• New code-object metadata — Objects/codeobject.c and Include/cpython/code.h.
• Argument scoping rules — Python/symtable.c.
• Future statement flags — Python/future.c and Lib/__future__.py.