PromQL

Active contributors: roidelapluie, codesome, krajorama, bwplotka

Purpose

PromQL is the query language used by Prometheus, Grafana, and downstream systems. The promql/ directory implements its lexer, parser, AST, and evaluation engine. The engine is purely a library — it doesn't open files or sockets; its inputs are a storage.Queryable and a query string, its output is a Result.

Directory layout

promql/
├── engine.go               # The query engine (~4,700 lines).
├── functions.go            # PromQL function implementations (~88,000 bytes; ~80 functions).
├── value.go                # Vector, Matrix, Scalar, String result types.
├── quantile.go             # Histogram quantile and friends.
├── durations.go            # promql-duration-expr parsing.
├── histogram_stats_iterator.go  # Stats over native histograms.
├── info.go                 # Implementation of the `info()` function.
├── query_logger.go         # Per-query log writer.
├── parser/
│   ├── generated_parser.y       # goyacc grammar (~56 KB).
│   ├── generated_parser.y.go    # generated parser.
│   ├── lex.go                   # Hand-written lexer.
│   ├── ast.go                   # AST node types.
│   ├── parse.go                 # Parser entry points.
│   ├── printer.go               # AST -> string.
│   ├── prettier.go              # PromQL pretty-printing.
│   ├── functions.go             # Function arity/type metadata.
│   └── posrange/                # Source position tracking.
└── promqltest/             # The PromQL test framework + golden test files.

How a query is evaluated

graph TD
    A[Query string] --> B[Lex]
    B --> C[Parse to AST]
    C --> D[Walk AST<br/>find selectors]
    D --> E[Querier per timerange]
    E --> F[Postings for matchers]
    F --> G[Open chunk iterators]
    C --> H[Engine.exec]
    H --> I[Eval at each step]
    G --> I
    I --> J[Result: Matrix/Vector/Scalar/String]

The engine has two execution modes:

• Instant query (NewInstantQuery) — one timestamp; returns a Vector (or scalar/string).
• Range query (NewRangeQuery) — start, end, step; returns a Matrix.

For both, the engine walks the AST top-down, opening queriers and chunk iterators lazily. The "step invariant" optimisation (#15094) caches sub-expressions whose result does not depend on t.

Sub-pages

• Parser and AST — the lexer, the goyacc grammar, and the AST walking utilities.
• Engine and evaluation — how the engine plans and runs queries, and how to add new functions.

Result types

Defined in promql/value.go:

Type	What it represents
Vector	Set of samples at one timestamp, one per series.
Matrix	A vector per series across many timestamps.
Scalar	A single (t, v) pair.
String	A single string value (e.g. topk(0) result).

Sample and Series carry both float values and native histograms; iterators differentiate via the chunkenc.ValueType enum.

Engine options

promql.EngineOpts (engine.go) exposes the knobs:

• MaxSamples — caps total samples touched by a query (--query.max-samples).
• Timeout — per-query timeout (--query.timeout).
• LookbackDelta — staleness window (--query.lookback-delta, default 5 min).
• NoStepSubqueryIntervalFn — fallback step for unbounded subqueries.
• EnableAtModifier, EnableNegativeOffset — toggles for legacy compatibility.
• EnablePerStepStats — exposes totalQueryableSamples per step (gated by --enable-feature=promql-per-step-stats).
• EnableExperimentalFunctions (gated by --enable-feature=promql-experimental-functions).
• EnableTypeAndUnitLabels, LookbackDelta.

Self-metrics

All metrics are prefixed prometheus_engine_*:

• prometheus_engine_query_duration_seconds (histogram, with slice label: queue|prepare|inner_eval|result_sort).
• prometheus_engine_queries_total.
• prometheus_engine_queries (gauge of currently running queries).
• prometheus_engine_query_log_failures_total.
• prometheus_engine_queries_concurrent_max and _canceled (gauges).

Concurrency

The engine implements a query gate (util/gate) limiting concurrent queries to --query.max-concurrency. Each query runs on its own goroutine, with sub-task fanout for binary operators.

Tracing

Spans are emitted for query_eval_total, query_inner_eval, and around chunk fetching. The trace ID is logged in the query log when tracing is enabled (3.11, #18189).

Integration points

• Storage: the engine takes a storage.Queryable and a storage.ChunkQueryable. Both interfaces live in storage/interface.go.
• Web API: web/api/v1.api.queryRange and api.query are the HTTP entrypoints.
• Rules: rules/manager.go::EngineQueryFunc wraps the engine for rule evaluation.
• promtool: cmd/promtool/query.go uses the engine indirectly via the HTTP API; cmd/promtool/unittest.go uses it directly.
• UI: the editor in web/ui/mantine-ui/src/promql/ uses the same parser via the npm @prometheus-io/codemirror-promql module.

Entry points for modification

• New PromQL function: add the implementation in promql/functions.go, register its arity in promql/parser/functions.go, regenerate the UI helpers via make generate-promql-functions, and add tests in promql/promqltest/testdata/.
• New operator or syntax: edit promql/parser/generated_parser.y, run make parser, then implement the AST node and engine logic.
• Engine optimisation: the engine has plenty of unused-sample-elimination room. New work should include a BenchmarkRangeQuery run before/after.
• Annotation (warning/info): add to util/annotations/. Annotations propagate through Result.Warnings and Result.Infos and surface in the UI.