Parser

The parser turns SQL text into an abstract syntax tree (AST). Everything else in the SQL pipeline starts from the parser's output.

Purpose

pkg/parser/ is a self-contained Go module (pkg/parser/go.mod) implementing a yacc-driven SQL parser plus the AST type system. It is shared between the TiDB server and many downstream tools (BR, Lightning, third-party tools that use github.com/pingcap/tidb/pkg/parser as a library).

Directory layout

pkg/parser/
├── parser.y              # Bison/yacc grammar (≈ 421k lines source)
├── parser.go             # Generated parser table (≈ 8 MB)
├── hintparser.y          # Optimizer hint grammar
├── hintparser.go         # Generated
├── lexer.go              # Hand-written lexer
├── digester.go           # SQL fingerprint / digest
├── keywords.go           # Reserved word tables
├── misc.go               # Lexer helpers, character-class tables
├── yy_parser.go          # Public Parser type and Parse* entry points
├── ast/                  # AST node definitions
├── auth/                 # Identity model (HostName, AuthSpec)
├── charset/              # Charset and collation registry
├── format/               # AST → string formatter
├── mysql/                # MySQL constants, error codes, type codes
├── opcode/               # AST operator codes
├── terror/               # Parser-domain errors
├── test_driver/          # Standalone driver for parser tests
├── duration/             # MySQL duration parsing
├── tidb/                 # TiDB-only AST extensions
├── types/                # SQL type system shared with the rest of the DB
└── util/                 # Misc helpers

Key abstractions

Type / function	File	Purpose
parser.Parser	pkg/parser/yy_parser.go	The yacc-generated parser wrapped in a usable Go type. Created with parser.New().
Parser.ParseSQL(ctx, sql, ...)	pkg/parser/yy_parser.go	Parses a multi-statement SQL string.
Parser.ParseOneStmt(sql, charset, collation)	pkg/parser/yy_parser.go	Parses exactly one statement, returns an ast.StmtNode.
ast.Node, ast.StmtNode, ast.ExprNode	pkg/parser/ast/	Top-level AST interfaces. Visit/restore methods on every node.
Scanner	pkg/parser/lexer.go	Hand-written lexer feeding the parser.
digester.NewDigester / Normalize	pkg/parser/digester.go	Produces a normalized "fingerprint" of a SQL string for use as the SQL digest.
mysql package	pkg/parser/mysql/	MySQL type codes, charset IDs, error numbers used everywhere.
charset.GetCollationByName, charset.GetCharsetInfo	pkg/parser/charset/	Charset/collation lookup.

How it works

graph LR
  text[SQL text] --> lex[Scanner<br/>lexer.go]
  lex --> yacc[Generated parser<br/>parser.go]
  yacc --> ast[AST tree<br/>ast/*]
  text --> dig[Digester<br/>digester.go]
  dig --> hash[SQL digest]

1. Callers call parser.New().ParseOneStmt(sql, charset, collation) (or the multi-statement variant).
2. yy_parser.go configures the scanner with the input and calls into parser.go's yacc-generated state machine.
3. The scanner (lexer.go) returns tokens; the parser builds AST nodes in pkg/parser/ast/.
4. The result is an ast.StmtNode (or []ast.StmtNode for ParseSQL) whose concrete types live under pkg/parser/ast/ (e.g., ast.SelectStmt, ast.InsertStmt, ast.AlterTableStmt).
5. Independently, digester.go produces a normalized digest of the SQL text for slow-log correlation, plan caching, and statement summary.

AST node organisation

The pkg/parser/ast/ directory groups nodes by category:

File	Nodes
dml.go	SelectStmt, InsertStmt, UpdateStmt, DeleteStmt, Join, TableSource
ddl.go	CreateTableStmt, AlterTableStmt, DropTableStmt, partition specs
expressions.go	Operator nodes, BinaryOperationExpr, IsNullExpr, etc.
functions.go	FuncCallExpr, AggregateFuncExpr, WindowFuncExpr
misc.go	BeginStmt, CommitStmt, SetStmt, role/grant statements
flag.go	Per-node flag bits used for plan-replayer/redaction
format.go	Restore infrastructure for round-tripping AST → SQL

Every node implements two visitors: Accept(v Visitor) and Restore(ctx *RestoreCtx). The latter is used to render the AST back as SQL, which is essential for plan dumps, view rewriting, and several test pipelines.

Generated code

parser.go is generated from parser.y and is approximately 8 MB. Do not hand-edit it. To regenerate:

make parser

This runs goyacc (vendored under pkg/parser/goyacc/) on parser.y and hintparser.y. The Makefile targets parser_yacc and parser_fmt are part of make check.

Other generated artefacts:

• pkg/parser/generate_keyword/ produces the keyword classification tables.
• pkg/parser/test_driver/ is a small driver used from external *_test.go files to exercise the parser without pulling in the rest of TiDB.

Integration points

• Session (pkg/session/session.go) calls Parser.ParseSQL for each incoming query.
• Planner (pkg/planner/) consumes ast.StmtNode and never re-parses.
• Hints are parsed in two ways: in-line /*+ ... */ hints are handled inside the main parser and routed through hintparser; legacy tidb_hints uses the same hint parser.
• Plan cache keys on the SQL digest produced by digester.go.
• Slow log correlates entries by SQL digest.
• Stmt summary keys by digest plan + digest text.

Entry points for modification

Most contributors only touch the parser when adding a new statement, expression, or hint:

• New SQL syntax → edit pkg/parser/parser.y, add the AST node under pkg/parser/ast/, then make parser.
• New keyword → register in pkg/parser/keywords.go (and update reserved_words_test.go).
• New scalar function name → register in the planner/expression layers as well; the parser only needs to know it's a function.
• Hints → pkg/parser/hintparser.y plus the consumers in pkg/planner/core/hint_utils.go.

After any of those changes, make bazel_prepare is required (Bazel test targets reference parser files in their srcs lists).

For SQL-grammar reference docs the upstream end-user docs live at https://docs.pingcap.com/tidb/stable/sql-statement-overview; this page covers only the in-repo implementation.