Parser and AST

The PromQL parser is split into a hand-written lexer, a goyacc-generated grammar, and a small AST package that the rest of the codebase imports.

Files

promql/parser/
├── lex.go                  # Lexer (~32 KB).
├── lex_test.go             # Lexer test cases.
├── generated_parser.y      # goyacc grammar (~56 KB).
├── generated_parser.y.go   # generated parser, do not edit by hand.
├── parse.go                # ParseExpr / ParseMetric / ParseMetricSelector entry points.
├── ast.go                  # Expr / VectorSelector / MatrixSelector / Call / BinaryExpr / …
├── functions.go            # Function metadata: arity, return type, experimental flag.
├── features.go             # Feature gate helpers used by the parser.
├── printer.go              # AST -> canonical string.
├── prettier.go             # AST -> formatted multi-line string.
├── prettier_rules.md       # In-repo specification for the pretty-printer.
├── value.go                # Constants for `vector` / `matrix` / `scalar` / `string` value types.
└── posrange/               # Source position tracking for parse errors.

Lexer

lex.go is a hand-written scanner. It tracks paren depth, string quoting, comment state, and produces tokens with Pos (byte offset) and PosRange markers used by parser error messages.

Extend the lexer if you introduce a new keyword, operator, or duration suffix. The PromQL promql-duration-expr syntax (e.g. 5m + step()) was added by extending the lexer alongside the parser.

Grammar

generated_parser.y is a yacc grammar; the project uses goyacc (a port to Go of yacc) to generate generated_parser.y.go. Common extension paths:

• New function: functions are recognised generically — register the name + arity in promql/parser/functions.go::Functions.
• New keyword (e.g. bool): add a token, terminal, and reduction rule.
• New operator (e.g. </, >/): the trim-bucket operators were added in #17904.
• New modifier (e.g. info(...) annotation): modifier syntax in PromQL is added by introducing new alternatives in the binary expression rules.

To regenerate after editing:

make install-goyacc   # one-time, installs go install golang.org/x/tools/cmd/goyacc@v0.6.0
make parser

CI runs make check-generated-parser to ensure the committed .y.go matches the grammar.

When debugging the parser, set yyDebug = 1..5 and yyErrorVerbose = true near line 600 of the generated file (do not commit those changes).

AST node types

ast.go defines:

Type	What it represents
VectorSelector	up{job="foo"}[5m] offset 3m
MatrixSelector	A VectorSelector paired with a range duration.
SubqueryExpr	( <expr> )[5m:1m]
Call	rate(...), histogram_quantile(...), etc.
BinaryExpr	a + b, with vector matching modifiers (on, ignoring, group_left).
AggregateExpr	sum by (label) (...), topk(5, ...).
NumberLiteral / StringLiteral	Literals.
UnaryExpr	Unary minus.
ParenExpr	Explicit parentheses (preserved to keep printer output stable).
StepInvariantExpr	Engine-internal wrapper inserted by engine.go::PreprocessExpr.
DurationExpr	promql-duration-expr (3.10) — duration computed from sub-expressions.

The Walk helper in ast.go is the canonical way to traverse an AST. It is used by:

• engine.go — to find selectors and prepare queriers.
• web/api/v1.api.targetRelabelSteps and api.parseQuery.
• rules/manager.go — to compute series-set fingerprints for rule sequencing.
• cmd/promtool/analyze.go — for ad-hoc analysis.

Parser entry points

Function	Use
parser.ParseExpr(s)	Parse a full PromQL expression.
parser.ParseMetric(s)	Parse a single label-set literal (up{job="foo"}).
parser.ParseMetricSelector(s)	Parse a vector selector for use as a matcher.
parser.NewParser(opts).Parse(s)	Reusable parser; preferred when parsing many strings (avoids allocation churn).

Errors carry posrange.PositionRange for IDE-style highlighting.

Pretty printing

prettier.go produces multi-line, deterministically formatted PromQL. The rules are documented in prettier_rules.md. The web UI's "Format query" button calls /api/v1/format_query which uses Prettify.

Stability

The PromQL surface is the project's most-stable user-facing API. New syntax goes through one or more releases behind --enable-feature=promql-experimental-functions before being made unconditional.

Entry points for modification

• New AST node: add to ast.go, then handle it in printer.go, prettier.go, engine.go::Eval, and rules/manager.go::shouldRecord.
• New experimental function: mark the entry in functions.go with Experimental: true. The engine will refuse to call it unless EnableExperimentalFunctions is set.
• New annotation: define it in util/annotations/ so the parser, engine, and rules manager can emit consistent strings.

See Engine for how the parsed AST is executed.