Parsers

src/Parsers/ turns SQL text into an AST. The codebase ships two parser layers:

1. A hand-rolled recursive-descent parser for the dialect ClickHouse actually accepts. This is what runs in production.
2. An ANTLR-based grammar under utils/antlr/ for tooling and contributor experimentation. Not on the hot path.

This page covers the production parser.

Big picture

graph LR
    SQL[SQL string] --> Lexer[Lexer.cpp<br/>tokenize]
    Lexer --> Stream[Tokens stream]
    Stream --> Parser[ParserQuery / ParserSelectQuery / ...]
    Parser --> AST[AST tree<br/>ASTSelectQuery, ASTFunction, ASTLiteral, ...]

Key files

File	Purpose
src/Parsers/Lexer.cpp	Tokenizer. Emits Token enums (Number, StringLiteral, Quoted_identifier, BareWord, OpeningRoundBracket, Comma, etc.).
src/Parsers/IParser.h	Base parser combinator interface.
src/Parsers/IParserBase.h	Common helpers used by every concrete parser.
src/Parsers/parseQuery.cpp	Top-level entry point. Picks ParserQuery and runs it.
src/Parsers/IAST.h	Base AST node. Each node knows how to format itself back to SQL.
src/Parsers/ASTSelectQuery.cpp, ASTSelectWithUnionQuery.cpp, ASTInsertQuery.cpp, ASTAlterQuery.cpp, …	Concrete AST nodes.
src/Parsers/ParserSelectQuery.cpp	The SELECT parser.
src/Parsers/ParserCreateQuery.cpp	CREATE TABLE/DATABASE/VIEW/USER/....
src/Parsers/ParserAlterQuery.cpp	ALTER (a tiny DSL of its own).
src/Parsers/ParserExpression.cpp, ExpressionListParsers.cpp, ExpressionElementParsers.cpp	The expression parser.
src/Parsers/Access/, src/Parsers/MySQL/, src/Parsers/SubstraitProto/	Dialect-specific parsers (RBAC DDL, MySQL-compat parser used by the MySQL handler, Substrait protobuf importer).

How parsing works

Each parser is a class with a parseImpl(Pos &, ASTPtr &, Expected &) method. It reads tokens through Pos (a peekable iterator), assembles AST nodes, and either succeeds or rolls the position back. Parsers compose: ParserSelectQuery calls ParserExpressionList, which calls ParserExpressionWithOptionalAlias, which calls ParserExpression, etc.

The pattern looks like:

bool ParserSomething::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
{
    ParserKeyword keyword{Keyword::SELECT};
    if (!keyword.ignore(pos, expected))
        return false;
    // ... assemble AST ...
    node = std::make_shared<ASTSelectQuery>(...);
    return true;
}

Failures unwind by restoring pos. Errors are surfaced via Expected, which collects the alternatives that were tried at the failure site so the message reads "expected one of: SELECT, WITH, EXPLAIN".

Format and roundtripping

Every AST node implements formatImpl(...) and formatQueryImpl(...). This means an AST can be re-serialized to a canonical SQL string. clickhouse format and the in-memory rewriting passes (e.g. IdentifierAccessLister, DDLOnClusterRewriter) rely on this.

AGENTS.md notes the convention: write f rather than f() when referring to a function name in prose, since f() looks like an application. The parser preserves both forms — f is an identifier, f(...) is ASTFunction.

Specialised parsers

• MySQL parser (src/Parsers/MySQL/) — used by the MySQL wire handler to accept MySQL-specific syntax.
• Substrait reader (src/Parsers/SubstraitProto/) — converts Substrait plans into ASTSelectQuery. Used by analytics tooling.
• Access DDL (src/Parsers/Access/) — CREATE/ALTER/DROP USER/ROLE/QUOTA/SETTINGS PROFILE/... and GRANT/REVOKE are large enough to live in their own file set.
• MakeASTForLogicalFunction.h — programmatic builders used by analyzer passes when synthesizing AST fragments.

Token-level features

Lexer.cpp handles:

• C-style and double-dash comments.
• Backtick-, double-quote- and unquoted identifiers (with separate tracking of which one was used).
• Plain, escaped, hex, and bin string literals.
• Numbers in decimal, hex, octal, and floating point.
• Multi-character operators (<=>, ||, ->, …).
• Heredoc string literals $$ ... $$.

AGENTS.md reminds contributors that any literal SQL keyword in prose should be wrapped in inline backticks; the parser is what defines the canonical spelling of those keywords.

Errors and parsing limits

MAX_QUERY_SIZE (a setting) caps the size of the query text. Recursion-depth limits prevent stack overflows on adversarial input. Parser errors mention the line/column and the highest-priority alternatives.

Entry points for modification

• New keyword → add it to src/Parsers/Keywords.h.
• New SQL construct → write a ParserX class plus an ASTX and register the parser into the relevant container (e.g. ParserSelectQuery::parseImpl).
• New EXPLAIN mode → extend ASTExplainQuery and InterpreterExplainQuery.

Related pages

• Analyzer — what consumes the AST.
• Functions — the registry that resolves ASTFunction names.
• Apps → utilities — clickhouse format is a thin wrapper around the parser.