Debugging

What to reach for when something goes wrong, in dev or in production.

Logs

Prometheus uses log/slog via the wrapper in prometheus/common/promslog. Set the level with --log.level=debug|info|warn|error and the format with --log.format=logfmt|json. Per-component loggers are attached with slog.With(...) so log lines carry context like component=scrape-manager scrape_pool=node.

The cmd/prometheus/main.go wires a klogv1Writer to redirect Kubernetes client logs (klog v1 + v2) into the same stream — this is why some lines have a kubernetes-flavored prefix.

pprof and fgprof

The standard Go pprof endpoints are available at /debug/pprof/ (registered by web/web.go). On-demand wall-clock profiling via fgprof is at /debug/pprof/fgprof (added in 3.10, #18027). Common captures:

# CPU profile for 30s
go tool pprof http://localhost:9090/debug/pprof/profile?seconds=30

# In-use heap
go tool pprof -inuse_space http://localhost:9090/debug/pprof/heap

# Goroutine dump
curl http://localhost:9090/debug/pprof/goroutine?debug=2

# Wall-time (fgprof)
curl -o fgprof.pprof http://localhost:9090/debug/pprof/fgprof?seconds=30
go tool pprof fgprof.pprof

The bench_tsdb make target uses these profiles to produce SVG graphs after a write benchmark — see Makefile.

Runtime info

/api/v1/status/runtimeinfo returns startup time, GOMAXPROCS, GOMEMLIMIT, GC/debug environment, storage retention, and goroutine count. Built from cmd/prometheus/main.go's runtimeInfo() and the RuntimeInfo struct in web/api/v1/api.go.

/api/v1/status/buildinfo returns version, branch, commit, build-user, build-date, Go version. Driven by the prometheus/common/version package values populated at link time by promu.

/api/v1/status/flags and /api/v1/status/config echo the live flags and parsed config.

TSDB diagnostics

promtool tsdb analyze <data dir> [block]   # cardinality / chunk stats
promtool tsdb dump <data dir>              # dump samples as text
promtool tsdb list <data dir>              # block listing
promtool tsdb create-blocks-from openmetrics ...   # backfill

Inside the TSDB, the WAL replay status is exposed at /api/v1/status/walreplay while the server is starting. The corruptionCount counter in runtime info increments when the WAL repairs a corrupt segment (tsdb/repair.go).

Scrape debugging

The new UI's Targets page shows the last scrape error per target. The same data is at /api/v1/targets. The /api/v1/targets/relabel_steps endpoint returns the per-step relabel pipeline output for a target, useful when relabel rules are dropping things unexpectedly.

The --enable-feature=extra-scrape-metrics flag adds scrape_timeout_seconds, scrape_sample_limit, and scrape_body_size_bytes so you can spot targets close to their limits with PromQL.

When a scrape fails, the failure logger (scrape.FailureLogger, scrape/scrape.go) captures detail to a JSON file when --scrape.failure-log-file is set.

PromQL query debugging

promtool query instant <url> '<expr>' [time]
promtool query range <url> '<expr>' --start ... --end ... --step ...
promtool query series <url> --match='<matcher>'
promtool query labels <url>
promtool query analyze <url> --type=histogram --match='<matcher>'

The query log (--query.log-file=...) records every executed query along with timings produced by promql.Engine's engineMetrics. The promql/query_logger.go module owns the format.

For ad-hoc EXPLAIN-like work on a query, set yyDebug = 1..5 in promql/parser/generated_parser.y.go (after running make parser) to dump the parser's state. Don't commit that change — see CONTRIBUTING.md.

Tracing

If tracing.endpoint is set (config/config.go), Prometheus emits OTLP spans for queries, scrapes, and rule evaluations. The setup is in tracing/tracing.go. Common spans:

• query_range / query_instant — issued by web/api/v1/api.go.
• scrape — per scrape iteration.
• rule.exec — per rule invocation.

A traceID field is added to query log entries when tracing is enabled (fix in #18189 for 3.11).

Remote write troubleshooting

Key metrics emitted by the queue manager (storage/remote/queue_manager.go):

• prometheus_remote_storage_samples_pending
• prometheus_remote_storage_samples_failed_total
• prometheus_remote_storage_samples_retried_total
• prometheus_remote_storage_shards
• prometheus_remote_storage_sent_batch_duration_seconds
• prometheus_remote_storage_highest_timestamp_in_seconds

The shard count auto-scales every shardUpdateDuration (10s by default) using an EWMA with weight 0.2.

Notifier (Alertmanager dispatch) troubleshooting

prometheus_notifications_* metrics — set --log.level=debug to see batch rejections. Per-Alertmanager metrics gained an alertmanager dimension in 3.10 (#16355).

Kubernetes SD troubleshooting

prometheus_sd_kubernetes_* metrics expose Kubernetes API call counts and watch state. Deduplicated deprecation warnings (since 3.11) reduce log noise; if you see them flood again, that's a regression.

Common error catalogue

Symptom	Likely cause
out of bounds, out of order sample	OOO support disabled and timestamp older than head.maxt.
start timestamp out of order, ignoring	Two scrapes reported different start timestamps for a counter; benign in practice.
WAL corruption: ... followed by repair	Crash mid-write. tsdb/repair.go truncates and resumes; expect a corruptionCount bump.
Stuck walreplay on startup	Cluster of mmap chunks waiting; check disk IO. The --storage.tsdb.head-chunks-write-queue-size flag affects this.
EOF on remote-write target	TLS or proxy interrupting long-poll; tune keepalive_timeout/max_connections on receiver.
Memory blow-up after prepare for compaction	Aborted compaction left chunk and index buffers retained. Restart and watch prometheus_tsdb_compaction_*.

For deeper environmental debugging, the documentation/examples/remote_storage standalone binaries show how to write a minimal remote read/write target.