prometheus

Active contributors: roidelapluie, bjboreham, machine424, krajorama, bwplotka, aknuds1

Purpose

prometheus is the main server binary. It reads prometheus.yml, runs service discovery, scrapes targets, stores samples in the TSDB (or the agent WAL), evaluates recording and alerting rules, dispatches alerts to Alertmanagers, and serves the HTTP API and the React UI. A single instance is autonomous — Prometheus has no clustering or replication built in.

Directory layout

cmd/prometheus/
├── main.go                    # Entry point (~2,200 lines) — wires every subsystem.
├── features_test.go           # Tests for /api/v1/features.
├── main_test.go               # Wide-coverage e2e tests against a real server.
├── main_unix_test.go          # Unix-specific signal-handling tests.
├── main_upgrade_test.go       # End-to-end upgrade/downgrade tests.
├── query_log_test.go          # Query log behaviour.
├── reload_test.go             # SIGHUP reload coverage.
├── scrape_failure_log_test.go # Scrape failure logger.
├── upload_test.go             # `--write-documentation` and asset upload tests.
└── testdata/                  # Fixture configs and golden files.

How it starts up

main() in cmd/prometheus/main.go is the largest function in the codebase (~2,200 lines, the longest single Go source file by line count). It does these things in order:

1. Parse flags with kingpin into a flagConfig struct. Every flag's documentation is extracted by --write-documentation for docs/command-line/prometheus.md.
2. Detect agent mode from --agent.
3. Configure the runtime: automaxprocs for CPU, automemlimit for memory, slog for logging, and the klog v1/v2 redirect.
4. Build the storage stack: either a tsdb.DB (server mode) or a tsdb/agent.DB (agent mode), wrapped in a fanoutStorage that fans writes to the local DB plus any remote write queues registered via storage/remote.Storage.ApplyConfig.
5. Construct the managers: discovery.NewManager, scrape.NewManager, rules.NewManager, notifier.NewManager. None of them are started yet.
6. Build the PromQL engine: promql.NewEngine with Opts for max samples, lookback delta, query timeout, and per-step stats.
7. Build the HTTP API and UI: web.New(...) with options assembled from the flag config.
8. Wire reload: a SIGHUP handler and the /-/reload endpoint both call reload(...), which calls ApplyConfig on every manager.
9. Run everything via oklog/run.Group: a manager per group member with Execute/Interrupt. Closing any group member cancels everything.

Two operating modes

graph LR
    subgraph Server["Server mode (default)"]
        SDS1[discovery.Manager] --> SCM1[scrape.Manager]
        SCM1 --> ST1[fanoutStorage]
        ST1 --> TSDB[(tsdb.DB)]
        ST1 --> RW1[remote.QueueManager]
        RM1[rules.Manager] -->|reads| TSDB
        RM1 -->|alerts| NM1[notifier.Manager]
        WEB1[web.Handler] -->|queries| TSDB
    end

    subgraph Agent["Agent mode (--agent)"]
        SDS2[discovery.Manager] --> SCM2[scrape.Manager]
        SCM2 --> ST2[fanoutStorage]
        ST2 --> WAL[(tsdb/agent WAL)]
        WAL --> RW2[remote.QueueManager]
        WEB2[web.Handler<br/>limited routes]
    end

Server mode is the standard monitoring deployment. Agent mode is documented under Agent mode and forces enable-remote-write-receiver=false, disables the rule manager and query engine, and limits the API surface.

Key abstractions

Type / Function	File	Role
flagConfig	cmd/prometheus/main.go	All parsed CLI flags. Passed to subsystem option structs.
reload(...)	cmd/prometheus/main.go	Hot-reload entry point; called by SIGHUP and /-/reload.
loadConfig(...)	cmd/prometheus/main.go	Wraps config.Load with reload-failure metrics.
runtimeInfo()	cmd/prometheus/main.go	Populates web/api/v1.RuntimeInfo from current process state.
klogv1Writer	cmd/prometheus/main.go	Redirects klog v1 output into klog v2 / slog.

CLI flags

Every flag is registered on a kingpin parser. Notable ones:

• --config.file — path to prometheus.yml.
• --storage.tsdb.path / --storage.agent.path — data directory.
• --storage.tsdb.retention.time / --storage.tsdb.retention.size / --storage.tsdb.retention.percentage (3.11) — retention policies.
• --web.listen-address, --web.external-url, --web.route-prefix, --web.enable-lifecycle, --web.enable-admin-api, --web.enable-remote-write-receiver, --web.enable-otlp-receiver.
• --query.timeout, --query.max-concurrency, --query.max-samples, --query.lookback-delta, --query.log-file.
• --rules.alert.for-outage-tolerance, --rules.alert.for-grace-period, --rules.alert.resend-delay, --rules.max-concurrent-evals.
• --scrape.discovery-reload-interval, --scrape.timestamp-tolerance.
• --alertmanager.notification-queue-capacity, --alertmanager.drain-notification-queue-on-shutdown.
• --enable-feature=<comma-separated list> — see Feature flags.
• --agent — switch to agent mode.
• --log.level, --log.format — slog configuration.
• --web.config.file — path to a prometheus/exporter-toolkit/web config (TLS + basic auth).

The full list is regenerated to docs/command-line/prometheus.md via make cli-documentation.

Reload semantics

A reload (SIGHUP, /-/reload, or first config load) calls loadConfig -> cfg.Validate() -> for each manager, manager.ApplyConfig(cfg). If any ApplyConfig fails, the reload is aborted and the previous config remains in effect. Successful reloads update RuntimeInfo.LastConfigTime and ReloadConfigSuccess.

A reload cannot:

• Change --storage.tsdb.path / --storage.agent.path.
• Switch between server and agent mode.
• Add or remove TLS for the web listener (changes to --web.config.file are picked up on reload via the toolkit).

Integration points

main.go imports almost every package in the repository. Notable integrations:

• config.Load (config/config.go) parses prometheus.yml and is the source of truth for the rest of the program.
• discovery.NewManager plus _ "github.com/prometheus/prometheus/plugins" registers all enabled SDs (the import has the init() side effect; build tags determine which are included).
• scrape.NewManager is given an Appendable (server mode: the fanout storage; agent mode: the agent DB).
• rules.NewManager is constructed only outside agent mode. It's wired with the engine, the storage queryable, the notifier's Send method, and the web handler's templates.
• web.New registers the API handlers and serves embedded UI assets.

Entry points for modification

If you need to:

• Add a new CLI flag: add it next to similar flags in main.go, surface it through the relevant manager's Options, then run make cli-documentation.
• Change reload behaviour: find func reload( and add your manager to the slice. Returning an error from ApplyConfig will fail the reload.
• Add or change server endpoints: edit web/api/v1/api.go (or web/web.go for non-API routes), then update web/api/v1/openapi_paths.go and run the OpenAPI golden test.
• Add an SD plugin: register it via discovery.RegisterConfig in the SD's package, add a plugins/plugin_<name>.go file with the appropriate build tag, list it in MAINTAINERS.md, and add config tests under config/testdata/conf.good.yml.

See Subsystems for the libraries main.go consumes, Configuration for the YAML schema, and Web API for the HTTP surface.