How to monitor Prometheus

Prometheus exposes its own metrics for self-monitoring. This page lists the most useful ones, plus the Jsonnet mixin shipped in this repository.

/metrics endpoint

GET /metrics exposes the full set of self-metrics. They are namespaced by subsystem: prometheus_tsdb_*, prometheus_scrape_*, prometheus_engine_*, prometheus_rule_*, prometheus_notifications_*, prometheus_remote_storage_*, prometheus_sd_*, prometheus_http_*.

The Jsonnet mixin

documentation/prometheus-mixin/ ships an opinionated bundle of recording rules, alerting rules, and Grafana dashboards. It is the canonical "monitoring Prometheus with Prometheus" recipe. To consume:

go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
jb install github.com/prometheus/prometheus/documentation/prometheus-mixin
mixtool generate documentation/prometheus-mixin/mixin.libsonnet

The mixin emits two YAML files (alert rules + recording rules) and one or more Grafana dashboards. Maintainer: Matthias Loibl (@metalmatze).

Logging

Configure with --log.level=debug|info|warn|error and --log.format=logfmt|json. JSON is recommended for ingestion into log pipelines.

Key log fields:

• component= — the subsystem (scrape-manager, rule-manager, notifier, tsdb, web, …).
• caller= — file:line (for slog text mode).
• traceID= (when tracing is enabled) — added in 3.11 (#18189) so query log entries can be correlated with traces.

The query log (--query.log-file) and the scrape failure log (--scrape.failure-log-file) are JSONL files written by util/logging.JSONFileLogger. Both are reopened on SIGHUP for log rotation.

Metrics to watch

Scrape health

• up — per target. up == 0 is the universal "target is sick" signal.
• scrape_duration_seconds — per scrape latency.
• prometheus_target_scrape_pool_targets — current target count per pool.
• prometheus_target_sync_failed_total — scrape pool reload failures.

TSDB health

• prometheus_tsdb_head_series — total active series.
• prometheus_tsdb_head_samples_appended_total (rate) — append throughput.
• prometheus_tsdb_compactions_failed_total — non-zero is bad.
• prometheus_tsdb_wal_corruptions_total — non-zero means a WAL repair fired.
• prometheus_tsdb_data_replay_duration_seconds — startup replay time.
• prometheus_tsdb_head_active_appenders — concurrent appenders in flight.

Query engine health

• prometheus_engine_query_duration_seconds{slice} — per-slice (queue/prepare/inner_eval/result_sort) histogram.
• prometheus_engine_queries — currently running.
• prometheus_engine_queries_concurrent_max — gate ceiling.
• prometheus_engine_queries_canceled — non-zero means timeouts or context cancellation.

Rule evaluation

• prometheus_rule_evaluation_failures_total — non-zero requires investigation.
• prometheus_rule_group_iterations_missed_total — missed evaluations (system overloaded).
• prometheus_rule_group_last_evaluation_seconds — wall-clock of last eval per group.

Notifications

• prometheus_notifications_dropped_total — alerts that did not reach an Alertmanager.
• prometheus_notifications_queue_length / _capacity — queue depth per AM.
• prometheus_notifications_latency_seconds — round-trip to AM.

Remote write

• prometheus_remote_storage_samples_pending — backlog. Steady > 0 means the receiver is slow.
• prometheus_remote_storage_samples_failed_total{reason} — non-retryable failures.
• prometheus_remote_storage_shards{state} — shard auto-scaling state.
• prometheus_remote_storage_highest_timestamp_in_seconds — freshness signal.
• prometheus_remote_storage_sent_batch_duration_seconds — receiver latency.

Service discovery

• prometheus_sd_received_updates_total — SD activity.
• prometheus_sd_last_update_timestamp_seconds (3.11) — last time an SD pushed an update; alert if stale.
• prometheus_sd_kubernetes_*, prometheus_sd_consul_*, etc. — SD-specific.

Web layer

• prometheus_http_request_duration_seconds{handler} — per-route latency.
• prometheus_http_response_size_bytes{handler} — response sizes.
• prometheus_http_requests_total{code,handler} — error-rate signal.

Configuration health

• prometheus_config_last_reload_successful — alert on ==0.
• prometheus_config_last_reload_success_timestamp_seconds.

Process / runtime

• process_cpu_seconds_total, process_resident_memory_bytes, go_gc_duration_seconds, go_goroutines, process_open_fds.
• prometheus_build_info — labels include version, branch, revision (used in dashboards for legend).

Alerting

The prometheus-mixin ships these alerts (paraphrased):

• PrometheusBadConfig — config reload failed.
• PrometheusNotificationQueueRunningFull.
• PrometheusErrorSendingAlertsToSomeAlertmanagers.
• PrometheusNotConnectedToAlertmanagers.
• PrometheusTSDBReloadsFailing, PrometheusTSDBCompactionsFailing, PrometheusTSDBWALCorruptions.
• PrometheusRuleFailures.
• PrometheusMissingRuleEvaluations.
• PrometheusTargetLimitHit.
• PrometheusRemoteStorageFailures, PrometheusRemoteWriteBehind, PrometheusRemoteWriteDesiredShards.
• PrometheusOutOfOrderTimestamps, PrometheusErrorSendingAlertsToAnyAlertmanager.

The full set is in documentation/prometheus-mixin/alerts.libsonnet.

Tracing

When tracing.endpoint is configured, the server emits OTLP spans for queries, scrapes, rules, and remote-write batches. See Tracing for the spec.

Dashboards

The mixin exports Grafana dashboards (prometheus.json, prometheus-remote-write.json). They cover the same metric set as the alerts. Use them as a starting point for custom dashboards.