Engine and evaluation

promql/engine.go is the largest single Go file outside of test code (~4,700 lines). It owns the query lifecycle: parsing, planning, executing, and surfacing results.

Lifecycle of a query

sequenceDiagram
    participant Caller
    participant Engine
    participant Parser
    participant Q as storage.Querier
    participant TSDB

    Caller->>Engine: NewRangeQuery(qs, start, end, step)
    Engine->>Parser: ParseExpr(qs)
    Parser-->>Engine: AST
    Engine->>Engine: PreprocessExpr (step-invariant detection)
    Engine->>Engine: validate, allocate query state
    Engine-->>Caller: Query handle
    Caller->>Engine: q.Exec(ctx)
    Engine->>Engine: queue (gate.Start)
    Engine->>Q: Querier(mint, maxt)
    Engine->>Engine: walk AST step-by-step
    Q->>TSDB: chunks for the time range
    Engine-->>Caller: Result{Vector|Matrix|Scalar|String, Warnings, Infos}

The five lifecycle slices are tracked separately in prometheus_engine_query_duration_seconds{slice=...}:

Slice	What it measures
queue	Time spent waiting in the query gate.
prepare	AST validation, querier setup.
inner_eval	The recursive expression evaluation.
result_sort	Final stable sort of the result.

Key types

Type	Role
QueryEngine	Public interface used by callers (instant + range query construction).
Engine	Default implementation in engine.go.
Query	Per-query state (parsed AST, gate token, sample budget, lookback).
Result	Holds value + warnings + infos.
evaluator	Internal struct that walks the AST recursively; one per Exec call.
EvalNodeHelper	Per-step caches (matchers, label hashes, sub-results).
EngineOpts	Construction options.

evaluator.eval is a giant switch on AST node type. Each branch:

1. Recursively evaluates children.
2. Allocates output Vector/Matrix from pools.
3. Honours MaxSamples by counting via evaluator.samplesStats.

Step invariance

engine.go::PreprocessExpr walks the AST and wraps any subexpression whose value does not depend on the evaluation timestamp (e.g. constant scalars, @ modifiers, fully-fixed time ranges) with a StepInvariantExpr. The evaluator computes that subexpression once and reuses the result across all steps.

Sample budgets

Queries are bounded by MaxSamples. The evaluator increments samplesStats.IncrementSamplesAtTimestamp (promql/engine.go) on every fetched sample. When the budget is exceeded, the query aborts with errors.New("query processing would load too many samples").

The sample stats are also exposed via --enable-feature=promql-per-step-stats, surfacing totalQueryableSamples per step in the API response.

Functions

promql/functions.go defines around 80 PromQL functions. Each is a func(...) keyed by parser.Functions["name"]. Conventions:

• A function returning a Vector writes into a passed-in slice to allow pooling.
• Histogram-aware functions accept either chunkenc.ValueType and switch on the type.
• Experimental functions check EnableExperimentalFunctions via util/features.

Notable additions over the past year:

• histogram_quantiles(...) — variadic quantile (3.11, #17285).
• </, >/ — bucket trimming operators (3.11, #17904).
• fill(), fill_left(), fill_right() — fill-missing modifiers (3.10, #17644).
• info() — joins target_info data into a vector (3.x).
• first_over_time, ts_of_first_over_time — experimental aggregates (3.x).

Histograms in PromQL

Native histograms flow through the engine as *histogram.Histogram or *histogram.FloatHistogram values attached to Sample.H. Operations that produce floats from histograms (histogram_quantile, histogram_count, histogram_sum, …) live in quantile.go and info.go.

KahanAdd (util/kahansum/) is used in averaging to reduce numerical drift over long ranges; the 3.10 PR #18252 partially addressed a regression caused by switching to Kahan addition for histograms.

Vector matching

Binary operators support set-style matching:

• vector1 + vector2 — one-to-one matching on full label sets.
• vector1 + on (label) vector2 — match by listed labels.
• vector1 + ignoring (label) vector2 — match by all-but-listed labels.
• vector1 + on (label) group_left (extra) vector2 — many-to-one with auxiliary labels.

Implementation: evaluator.evalVectorMatch and helpers, including the recently optimised binopAndOrUnless paths from #17159 (3.11) which reduced heap allocations for and/or/unless.

Subqueries

SubqueryExpr evaluates the inner expression at the implicit interval, then treats the result as a Range for the surrounding [range:step] selector. The engine sets a fresh evaluator with the inner range and step.

Stats and tracing

stats.QueryStats (util/stats/) records:

• Total samples touched.
• Wall-time per slice.
• Per-step samples when enabled.
• Step-by-step storage seek counts.

These are surfaced in the API response when stats=all (or stats=full in older versions).

Query log

Setting --query.log-file=... enables a JSONL log: one entry per query with start time, duration, total queryable samples, total samples, peak samples, and the query text. The file is written by promql/query_logger.go. The query log is reopened on SIGHUP so log rotators work.

Concurrency model

The engine has a single gate that limits the number of concurrent queries; per-query parallelism is bounded by Go's scheduler and the query AST shape. Heavy queries do not preempt other queries except via the OS scheduler; this is by design.

Entry points for modification

• Add a function: edit functions.go for the implementation, parser/functions.go for the metadata, and add tests in promql/promqltest/testdata/<name>.test.
• Add an aggregator: aggregations have their own switch in evaluator.aggregation. Most existing ones are good templates.
• Optimise a path: start by writing a BenchmarkRangeQuery case for the offending shape and run with -benchmem. Reviewers expect benchstat output in the PR body.

See Parser for the AST and PromQL for the broader subsystem context.