Parser

The parser turns SQL text into a Query tree. It runs in two stages: a bison/flex grammar produces a raw parse tree, and parse analysis resolves names against the catalog and assigns types. Source: src/backend/parser/.

Directory layout

src/backend/parser/
├── analyze.c            # parse analysis: top-level dispatch
├── gram.y               # bison grammar (~18 KLOC, the largest file in the tree)
├── kwlist.h             # keyword list
├── parse_agg.c          # aggregate function checks
├── parse_clause.c       # FROM, WHERE, GROUP BY, ORDER BY, LIMIT
├── parse_coerce.c       # type coercion rules
├── parse_collate.c      # collation assignment
├── parse_cte.c          # WITH (Common Table Expressions)
├── parse_expr.c         # expression analysis (the bulk of the work)
├── parse_func.c         # function call resolution
├── parse_jsontable.c    # JSON_TABLE
├── parse_merge.c        # MERGE
├── parse_node.c         # node-construction helpers
├── parse_oper.c         # operator resolution
├── parse_param.c        # bind parameters ($1, $2, ...)
├── parse_relation.c     # range table, schema-qualified names
├── parse_target.c       # SELECT target list, INSERT VALUES
├── parse_type.c         # type-name resolution
├── parse_utilcmd.c      # DDL: CREATE TABLE, CREATE INDEX, etc.
├── parser.c             # entry: raw_parser, parse_analyze
├── scan.l               # flex scanner
├── scansup.c            # string handling helpers
└── parse_*.c            # ... a few more

Stage 1: raw parsing

raw_parser (in parser.c) calls into the bison-generated parser. The grammar is gram.y and the scanner is scan.l.

graph LR
    SQL[SQL string] --> Scanner["scan.l (flex)"]
    Scanner --> Grammar["gram.y (bison)"]
    Grammar --> RawTree[RawStmt list of Node*]

The output is a list of RawStmt nodes, each wrapping a SelectStmt, InsertStmt, CreateStmt, etc. Nothing has been semantically analyzed yet — names like foo.bar could be a column reference, a table-qualified column, or a record field; the raw tree just records the syntactic shape.

The grammar handles all SQL PostgreSQL recognizes: SELECT (with the full SQL standard plus extensions), INSERT/UPDATE/DELETE/MERGE, every DDL command, transactional commands, EXPLAIN, etc. The grammar file is by far the largest single source file in the tree at about 18 KLOC; it is regenerated by bison on build.

Adding a new keyword:

1. Add a token in gram.y (%token).
2. Add it to kwlist.h with the appropriate category.
3. Use it in production rules.
4. Run src/tools/check_keywords.pl to verify the keyword list and grammar agree.

Adding a new statement:

1. Define a new Node type in src/include/nodes/parsenodes.h.
2. Add grammar rules producing the new node.
3. Add a parse-analysis function (often in parse_utilcmd.c for DDL).
4. Add executor or utility-command code to actually run it.

Keywords

Keywords are partitioned into four reservation classes (in kwlist.h):

• UNRESERVED_KEYWORD — usable as identifiers in most contexts.
• COL_NAME_KEYWORD — usable as column names but not table aliases.
• TYPE_FUNC_NAME_KEYWORD — usable as type or function names.
• RESERVED_KEYWORD — never usable as an identifier.

Adding a reserved keyword breaks user code. The community is conservative — most new names go in as unreserved or column-name keywords.

Stage 2: parse analysis

parse_analyze (in analyze.c) takes a RawStmt and produces a fully typed Query. It dispatches based on statement type:

Query *
parse_analyze(RawStmt *parseTree, const char *sourceText, ...)
{
    ParseState *pstate = make_parsestate(NULL);
    /* configure pstate->p_sourcetext, etc. */
    Query *query = transformTopLevelStmt(pstate, parseTree);
    free_parsestate(pstate);
    return query;
}

Subroutines do the real work:

Function	Purpose
transformSelectStmt	Analyze a SELECT: range table, target list, WHERE, GROUP BY, etc.
transformInsertStmt	Analyze INSERT, including VALUES and INSERT...SELECT.
transformUpdateStmt, transformDeleteStmt, transformMergeStmt	Mutating statements.
transformCreateStmt	CREATE TABLE; resolves types, expands LIKE.
transformExpr	Recursive expression analysis (in parse_expr.c).
transformFromClause	FROM list: tables, joins, subqueries, lateral.

ParseState carries the surrounding context: the range table being built, parent parse states (for subqueries), LATERAL flag, hooks for plug-in resolution.

Range table

The output Query carries a range table (rtable) — a list of RangeTblEntry (RTE) structs, one per relation, subquery, function, values list, or CTE referenced in FROM. Range table entries are 1-indexed in Query nodes; the index is called the RTE index and shows up everywhere downstream.

Type coercion

The most subtle part of parse analysis is resolving the types of expressions. PostgreSQL has a complex but principled coercion system:

• Implicit casts between binary-coercible types (e.g., text and varchar).
• Assignment casts in INSERT/UPDATE target lists.
• Explicit casts via CAST(x AS type) or x::type.

Functions in parse_coerce.c and parse_type.c make these decisions. They consult pg_cast and pg_type for cast and type information.

Function and operator resolution

Overloading: PostgreSQL allows multiple functions and operators with the same name distinguished by argument types. Resolving a call goes through func_select_candidate (in parse_func.c) and oper_select_candidate (in parse_oper.c), which apply a documented multi-step disambiguation procedure (exact match → preferred-type match → ...).

Aggregates and window functions

parse_agg.c walks the post-analysis query and verifies that aggregate and window function calls obey their structural rules: aggregates can't appear in WHERE, GROUP BY columns must cover all non-aggregate target list columns, etc. It also classifies the query as a "grouping query" if any aggregate is present.

CTEs and subqueries

WITH clauses are handled in parse_cte.c. CTEs add to the parse state's CTE list; references in subsequent parts of the query resolve to those entries. CTEs can be recursive; parse_cte.c validates the recursion structure.

Subqueries are recursively analyzed via transformSelectStmt with a child ParseState whose parentParseState is the outer one — that's what enables outer-query references and LATERAL.

Output

The result of parse analysis is a Query (src/include/nodes/parsenodes.h):

typedef struct Query
{
    NodeTag        type;
    CmdType        commandType;     // CMD_SELECT, CMD_INSERT, ...
    QuerySource    querySource;     // QSRC_ORIGINAL, QSRC_PARSER, QSRC_INSTEAD_RULE, ...
    List          *rtable;          // range table
    FromExpr      *jointree;        // FROM clause as a tree
    List          *targetList;      // SELECT target list / SET clauses
    Node          *havingQual;
    List          *groupClause;
    List          *windowClause;
    List          *sortClause;
    Node          *limitOffset;
    Node          *limitCount;
    /* ... many more fields ... */
} Query;

This Query flows next into the rewriter (src/backend/rewrite/), which applies view rules and INSTEAD OF triggers, then into the planner (Planner).

Utility statements

Statements that aren't queries — DDL, transaction control, COPY, EXPLAIN — return a Query of command type CMD_UTILITY whose utilityStmt field carries the original parse-tree node. The planner is essentially a pass-through for these; the executor's ProcessUtility path dispatches them.

Entry points for modification

• New SQL syntax: edit gram.y, parsenodes, parse-analysis transforms; update keyword list if you added a keyword.
• New built-in function or operator: usually no parser change needed — register it in pg_proc.dat/pg_operator.dat and the existing resolution code finds it.
• New cast: pg_cast.dat plus an implementation function. Implicit casts are powerful — they affect resolution everywhere — and the project is conservative about adding new ones.

For what happens after parse analysis, see Planner. For the rule-based view rewriter, look at src/backend/rewrite/rewriteHandler.c.