Agent mode

Agent mode runs a stripped-down Prometheus that scrapes targets and ships every sample via remote write, with no local query engine and only a WAL-based store on disk. It is intended for edge or replicated collection where queryable data lives somewhere central (Mimir, Thanos, Cortex, or a vendored backend).

Invocation

prometheus --agent --config.file=/etc/prometheus/agent.yml --storage.agent.path=/data

--agent is a server-wide switch evaluated very early in cmd/prometheus/main.go. It changes:

• The storage from tsdb.DB to tsdb/agent.DB (tsdb/agent/db.go).
• The available web routes — see web/web.go::wrapAgent and the oldUIReactRouterPaths allowlist.
• The disabled subsystems: rules.Manager, promql.Engine, the read handler, the federation endpoint.

Storage

tsdb/agent.DB keeps a Head plus the WAL (tsdb/wlog/) but never compacts to blocks. Truncation is age-based — controlled by --storage.agent.retention.min-time (default 5 min) and --storage.agent.retention.max-time (default 4 h). Any sample older than the truncation horizon is dropped.

Agent series are retained as long as remote write watchers haven't caught up. Agent-mode-specific append paths:

• tsdb/agent/db_append_v2.go for the V2 appender.
• Custom WAL truncation that preserves records still pending in the watcher.

Configuration restrictions

The agent forbids:

• rule_files — no rule evaluation.
• alerting: — no alert dispatch (Alertmanager isn't reachable from queries that don't exist).
• remote_read: — querying isn't possible.

It requires at least one remote_write: entry (otherwise samples have nowhere to go).

The remote write configuration applies the same set of options as in server mode; the only practical change is that the WAL watcher is the only source of samples.

Use cases

• Edge collection: thousands of small Kubernetes clusters, each running an agent, all sending to a central Mimir.
• Multi-tenant collection: per-tenant agents pre-relabelling samples before shipping.
• Resource-constrained nodes: the WAL-only model uses much less memory and CPU than a full TSDB.

Self-metrics

Agent mode exposes the same prometheus_remote_storage_* metrics. Some prometheus_tsdb_* metrics are present but represent the agent's WAL-only behaviour:

• prometheus_agent_active_series — active series in the agent's head.
• prometheus_agent_truncate_duration_seconds — duration of WAL truncations.
• prometheus_agent_corruptions_total — same semantics as the server.

Key files

File	Role
tsdb/agent/db.go	Top-level agent DB (~1,400 lines).
tsdb/agent/db_append_v2.go	V2 appender for the agent.
tsdb/agent/series.go	Active-series tracker.
tsdb/agent/checkpoint.go	WAL checkpoint logic specific to age-based truncation.
cmd/prometheus/main.go	The --agent switch and per-mode wiring.

Recent fixes

• 3.11 (#17538) — fixed a memory leak from duplicate SeriesRefs being loaded as active series.
• 3.11 (#17948) — checkpoint policy is now driven by series-in-memory rather than time only.
• 3.9.1 (#17802) — fixed a crash shortly after startup due to invalid object types.

Entry points for modification

• New WAL record: add to tsdb/record/record.go and update both tsdb/agent/db.go::loadWAL and the watcher.
• Truncation policy: tsdb/agent.DB.gc() and truncate().
• New observable: add metrics in tsdb/agent/db.go near the existing ones; namespace prometheus_agent_*.

See Remote write for the dispatch path and TSDB / WAL for the underlying log machinery.