neondatabase/neon

Metrics

Neon exposes Prometheus metrics over HTTP from every service. The metrics client is libs/metrics/, a thin wrapper over the prometheus crate that adds a few convenience macros and the project's preferred bucket sets.

Where to scrape

Service	Default endpoint
Pageserver	`:9898/metrics` (configurable)
Safekeeper	`:7676/metrics`
Storage broker	gRPC port — minimal metrics; expose via separate axum endpoint when configured
Storage controller	`:1234/metrics`
Storage scrubber	(offline, no `/metrics`)
Proxy	`:7001/metrics` (configurable)
compute_ctl	per-compute HTTP port
Endpoint storage	configurable HTTP port

In cargo neon development the ports are visible in .neon/config.

Naming

Snake-case names, prefixed by service: pageserver_*, safekeeper_*, proxy_*, storcon_*, compute_*.
Suffix indicates type: _total (counter), _seconds (histogram, units of seconds), _count and _sum come automatically from histograms, _bytes, _count.
Exemplary metric: pageserver_smgr_query_seconds{request_type="GetPageAtLsn", tenant_id="..."} — histogram of GetPage@LSN latency.

High-value metrics

The four big files where metrics are defined:

pageserver/src/metrics.rs (~170 KB; the largest file in the project after tenant.rs)
safekeeper/src/metrics.rs (~37 KB)
proxy/src/metrics.rs (~23 KB)
storage_controller/src/metrics.rs (~16 KB)

A non-exhaustive guide to the most useful ones:

Pageserver

Metric	Type	What it tells you
`pageserver_smgr_query_seconds`	histogram	`GetPage@LSN` latency. Primary read SLO.
`pageserver_smgr_query_count_total`	counter	Request rate.
`pageserver_layer_count`	gauge	Per-tenant layer count. Spikes ⇒ stalled compaction.
`pageserver_resident_physical_size_bytes`	gauge	Local disk used per tenant.
`pageserver_remote_physical_size_bytes`	gauge	Remote storage used per tenant.
`pageserver_walredo_seconds`	histogram	Latency of one WAL-redo round trip.
`pageserver_walredo_replay_records_total`	counter	Records replayed (a high rate per request signals a deep redo chain).
`pageserver_remote_*_latency_seconds`	histogram	S3/Azure/GCS I/O latency.
`pageserver_compaction_seconds`	histogram	Compaction round duration.
`pageserver_evictions_with_lower_residence`	counter	Eviction churn.

Safekeeper

Metric	Type	What it tells you
`safekeeper_flush_lsn`	gauge	Per-timeline disk-durable LSN.
`safekeeper_commit_lsn`	gauge	Per-timeline quorum-confirmed LSN.
`safekeeper_active_segments`	gauge	Active timelines.
`safekeeper_election_count_total`	counter	Election starts; sustained rate ⇒ instability.
`safekeeper_wal_storage_*`	various	Disk usage and segment counts.
`safekeeper_wal_backup_lag_lsn`	gauge	How far behind remote-storage WAL backup is.

Proxy

Metric	Type	What it tells you
`proxy_connection_total`	counter	Connection attempts by protocol and outcome.
`proxy_compute_connection_seconds`	histogram	Time spent acquiring a compute backend.
`proxy_io_*`	counter	Bytes / messages forwarded.
`proxy_console_request_*`	various	Console API call rate / latency.

Storage controller

Metric	Type	What it tells you
`storcon_pageservers_alive`	gauge	Heartbeating pageserver count.
`storcon_reconciles_*`	counter / histogram	Reconciler iterations and duration.
`storcon_tenant_shards_total`	gauge	Total shards under management.

Cardinality discipline

The codebase is careful about labels. The general rule:

OK on hot-path counters/histograms: request_type, status_code, http_method.
NOT OK: tenant_id, timeline_id, key (would create unbounded series).
OK on coarse gauges: tenant_id (the pageserver_resident_physical_size_bytes family is per-tenant and reviewed for cardinality cost).

When in doubt, look at how nearby metrics in the same metrics.rs file are labeled.

Histogram buckets

Every histogram has explicit buckets tuned for the metric's expected range. Don't accept prometheus's default buckets — for sub-millisecond latencies they are useless.

Examples:

pageserver_smgr_query_seconds: buckets in 1 ms increments up to 100 ms, then 100/200/500/1000/… up to 60 s.
pageserver_remote_*_latency_seconds: 10 ms .. 30 s.
proxy_compute_connection_seconds: 50 ms .. 30 s.

Histograms vs summaries

The project uses histograms, not summaries. This matches Prometheus best practice and makes cross-instance aggregation work.

Adding a new metric

// in <service>/src/metrics.rs
use metrics::{register_int_counter, IntCounter};
use once_cell::sync::Lazy;

pub static MY_METRIC: Lazy<IntCounter> = Lazy::new(|| {
    register_int_counter!(
        "service_my_metric_total",
        "What the metric counts."
    ).unwrap()
});

Then MY_METRIC.inc() from the call site. For histograms, prefer Histogram::observe(seconds) over manual buckets.