Rules

Active contributors: roidelapluie, bjboreham, codesome

Purpose

The rules/ package evaluates recording rules (precomputed PromQL whose result is stored back as a series) and alerting rules (PromQL whose nonzero result becomes an alert). Rule files are loaded from disk at startup and on reload, grouped by file, and evaluated on a per-group interval.

Directory layout

rules/
├── manager.go     # Top-level rules.Manager: loads files, owns groups.
├── group.go       # Group: ordered set of rules, schedule, dependency analysis.
├── rule.go        # The Rule interface; Common helpers.
├── recording.go   # RecordingRule type + evaluation.
├── alerting.go    # AlertingRule type, alert state machine, restoration.
├── origin.go      # Wraps spans with rule attribution metadata.
└── fixtures/      # Sample rules for tests.

Group lifecycle

sequenceDiagram
    participant M as Manager
    participant G as Group
    participant E as promql.Engine
    participant Q as Queryable
    participant S as Storage
    participant N as notifier.Manager

    Note over M: ApplyConfig parses rule files
    M->>G: NewGroup(ruleFile, opts)
    G->>G: tick(interval)
    loop every interval
        G->>E: NewInstantQuery(rule.expr, t)
        E->>Q: Query series
        E-->>G: Vector
        alt RecordingRule
            G->>S: appender.Append(metric, t, v)
        else AlertingRule
            G->>G: state machine (inactive -> pending -> firing)
            G->>N: Send([]Alert)
        end
    end

Group.run is the per-group goroutine. It computes the next evaluation time using deterministic offset (so multiple HA replicas evaluate at the same wall-clock time) and runs the rules in declaration order. Rule dependencies are detected up-front by walking the AST (Group.dependencyGraph), so independent rules can be evaluated concurrently.

Key types

Type / Function	File	Role
Manager	rules/manager.go	Reload, file -> group registration, metric exposition.
Group	rules/group.go	Owned execution unit; one ticker, one storage appender per iteration.
Rule	rules/rule.go	Interface implemented by RecordingRule and AlertingRule.
RecordingRule	rules/recording.go	Stores result as __name__=record_name.
AlertingRule	rules/alerting.go	Alert state machine; emits ALERTS{...} and ALERTS_FOR_STATE{...} series.
EngineQueryFunc	rules/manager.go	Adapts promql.Engine -> QueryFunc.
ManagerOptions	rules/manager.go	Options bag: engine, queryable, notifier.Send, external labels, etc.

Alert state machine

stateDiagram-v2
    [*] --> Inactive
    Inactive --> Pending: expr non-empty
    Pending --> Firing: held > FOR
    Firing --> Inactive: expr empty
    Pending --> Inactive: expr empty before FOR

Alerts include their evaluation timestamp, labels, annotations, and the ALERTS{alertstate=...} indicator series.

The for duration is honoured to avoid alert flap on transient violations. State is restored from disk on startup using the ALERTS_FOR_STATE series — when Prometheus has been down longer than --rules.alert.for-outage-tolerance (default 1h), the timer resets.

The 3.11 fix #18244 corrects state restoration when a rule's for value is increased in the config file: before the fix, the alert would wrongly reset to pending; after, the previously-firing alert continues firing.

Notification dispatch

When an alerting rule fires (or resolves), the group calls ManagerOptions.NotifyFunc(alerts). In cmd/prometheus/main.go, this is wired to notifier.Manager.Send. See Notifier for the dispatch path.

--rules.alert.resend-delay (default 1m) caps how often the same firing alert is re-sent.

Concurrent group evaluation

Within a group, rules can be evaluated concurrently when their dependency graph permits. This is bounded by --rules.max-concurrent-evals and a golang.org/x/sync/semaphore. Across groups, each group has its own goroutine.

The dependency analysis walks the AST and treats any rule whose result is referenced by another rule's expression as a dependency. Groups whose rules are interdependent fall back to sequential execution.

External labels

rules.ExternalLabels (set from global.external_labels in prometheus.yml) are appended to every alerting rule's labels before sending to the notifier. They are not applied to recording-rule output.

global.external_url is templated into alert annotations with the $externalURL template variable (see template/).

Templating

Alert labels and annotations support Go-template syntax with helpers from template/template.go:

• value, humanize, humanizePercentage, humanizeDuration, humanizeTimestamp.
• query (run a sub-query against the engine).
• safeHtml, reReplaceAll, match, title, toLower, toUpper.

Templates run with the rule's evaluation time set as the current sample time. The 3.11 bugfix #18375 ensures alert state is correctly restored when labels themselves contain template expressions.

Self-metrics

• prometheus_rule_group_iterations_total / _missed_total
• prometheus_rule_group_duration_seconds
• prometheus_rule_group_last_evaluation_seconds
• prometheus_rule_group_rules (gauge)
• prometheus_rule_evaluation_failures_total
• prometheus_rule_evaluation_duration_seconds
• prometheus_alerting_rule_tracker_alerts (firing/pending/inactive counts)

Integration points

• Storage: uses a Queryable for evaluation and an Appendable for recording rule output.
• PromQL: every evaluation goes through the engine.
• Notifier: uses notifier.Manager.Send for alert dispatch.
• HTTP API: /api/v1/rules and /api/v1/alerts use the manager's accessors via web/api/v1.RulesRetriever.
• promtool: promtool test rules reuses Group and Rule directly to provide a unit-test runner.

Entry points for modification

• New rule type: add a struct implementing Rule, an Eval(ctx, t) method, and a Type() returning a unique tag. Most existing helpers in manager.go operate on the interface, so extension is mechanical.
• New evaluation strategy: the EvalIterationFunc hook in Group.run allows a custom mode (e.g. dry-run for promtool test rules).
• Tweak the alert state machine: consult rules/alerting_test.go first — the "alert restoration" test set is large and intentionally pedantic.

See Notifier for what happens after a firing alert is produced and Templates for the templating helpers.