Design decisions

A handful of recurring decisions that the project enforces deliberately and that often surprise new contributors.

A single, autonomous server

Prometheus has no built-in clustering, replication, or shared storage. Each instance is autonomous: it scrapes, stores, and queries on its own. The README is explicit:

No dependency on distributed storage; single server nodes are autonomous.

The trade-off: HA / global visibility / long-term storage are someone else's problem (Mimir, Thanos, Cortex, Grafana Cloud, etc.). The benefit: every operator can run a useful Prometheus from a single binary, no sidecars needed. This decision shapes the storage interface (one local + N remotes), the lack of a leader-election library, and the per-instance assumption baked into every subsystem.

Pull-based scraping

Prometheus pulls metrics over HTTP rather than receiving them via push. The reasoning, summarised from the FAQs and docs/getting_started.md:

• Scrape configs are versioned alongside Prometheus.
• Targets don't need to know where Prometheus is.
• Service discovery is naturally pull-friendly.
• Failed targets are obvious (up=0); push systems hide failures.

For genuinely batch-job workloads, the Pushgateway intermediary exists. For OpenTelemetry / push agents, the OTLP receiver exists. Neither replaces the canonical pull model.

File-based configuration

prometheus.yml is the source of truth. There is no admin API to mutate scrape configs at runtime. The 3.x feature flag system added a registry but did not change this — every behaviour change still has a YAML or CLI representation.

The reasoning: declarative config is auditable, version-controllable, and reload-safe. Imperative APIs would reinstate the etcd problem the project deliberately avoided in v1.

Append-mostly TSDB

The TSDB is optimised for append-mostly workloads. Out-of-order ingestion exists but is bounded by out_of_order_time_window. Updates and deletes go through tombstones, not in-place mutation. Compaction is offline (a separate goroutine) so the append path stays predictable.

The Gorilla XOR (and now XOR2) chunk encodings are tuned for this exact pattern: same-value samples take 1 bit, small-delta samples take few bytes.

Goroutine-per-thing

Many subsystems own a goroutine per logical unit:

• One per scrape target.
• One per rule group.
• One per WAL watcher subscriber.
• One per Alertmanager send loop (3.10+).

This is intentional. Go's scheduler is cheap, and per-target isolation means a slow target doesn't stall its peers. The downside is a per-goroutine fixed cost — at 100k targets the goroutine count gets non-trivial. Performance work has cycled back to consolidate where the per-target model is genuinely overkill (e.g. the unified WAL watcher, the consolidated discovery manager).

Build tags everywhere

The codebase uses build tags pervasively:

• Three labels representations (slicelabels, stringlabels, dedupelabels) — pick the memory/CPU trade-off at compile time.
• TSDB forcedirectio for direct-IO mode.
• builtinassets to embed the UI.
• remove_all_sd + enable_<sd>_sd for slim builds.

The rationale is a single binary that can be tuned at compile time without runtime indirection. CI runs every meaningful tag combination so the matrix stays valid.

Stable HTTP API

The /api/v1/... route set has not had a breaking change since v2.0.0 (Nov 2017). New routes are added; existing routes only get additive changes. This is enforced by the OpenAPI golden test (web/api/v1/openapi_golden_test.go).

Six-week release cadence

Releases ship every six weeks regardless of how much landed. The release coordinator role rotates. Predictability is favoured over big-bang releases — features that aren't ready get held over to the next cycle.

Performance is a first-class review concern

Every PR touching a hot path needs a benchmark and benchstat numbers. AGENTS.md is explicit. Maintainers will ask for prombench runs for changes that affect the TSDB, scrape, or remote write. The cost: longer review cycles for performance-adjacent work; the benefit: 13 years of compounded performance gains.

"No third-party agreement, no SD"

discovery/README.md puts a hard rule on new SDs: an SD mechanism must be reasonably well-established and used across multiple organisations, must have a committed maintainer with push access, and cannot be a brand-new variant of an existing mechanism. Several proposed SDs have been rejected for not meeting this bar.

The principle: one stable, generic mechanism (file_sd + http_sd) for everything custom; native plugins for systems with broad ecosystem adoption.

Don't break the storage API quietly

Storage interface changes are coordinated in tracking issues (#17632 for AppenderV2). The tsdb/CHANGELOG.md is maintained separately from the project CHANGELOG. The on-disk block format must remain readable across minor releases.

This is why the AppenderV1 -> V2 migration has taken multiple releases: every consumer (TSDB, agent, scrape, OTLP receiver, remote write receiver, teststorage) must implement V2 before V1 can be removed.