Metrics

etcd exposes Prometheus metrics on /metrics at the client port. Every subsystem owns its own metrics.go:

• server/etcdserver/metrics.go — server-level: leader changes, proposals, applied index, lag.
• server/etcdserver/api/etcdhttp/metrics.go — HTTP layer.
• server/etcdserver/api/v3rpc/metrics.go — gRPC: etcd_grpc_* counters and histograms.
• server/etcdserver/api/rafthttp/metrics.go — peer transport.
• server/storage/backend/metrics.go — bbolt commit/defrag/snapshot.
• server/storage/mvcc/metrics.go — MVCC: revisions, watch counts, hash latency.
• server/storage/wal/metrics.go — WAL fsync.
• server/lease/metrics.go — leases granted/revoked/expired.
• server/auth/metrics.go — auth checks.
• server/etcdserver/apply/metrics.go — apply duration, failure counts.
• server/etcdserver/api/v3compactor/ — compaction outcomes.

The full inventory is regenerated by tools/etcd-dump-metrics/, which boots a member, scrapes /metrics, and prints the metric catalogue. CI uses it to detect accidental metric drops.

Headline metrics

Metric	Why it matters
etcd_server_has_leader	0 means the cluster has no leader. Page everyone.
etcd_server_leader_changes_seen_total	Frequent changes signal flapping.
etcd_server_proposals_failed_total	Increments when a write is rejected (quota, auth, slow apply).
etcd_server_proposals_pending	Sustained non-zero suggests apply backlog.
etcd_disk_wal_fsync_duration_seconds	The single most important latency series; SSDs should keep p99 < ~10 ms.
etcd_disk_backend_commit_duration_seconds	bbolt commit latency; shoots up if the database file is fragmented.
etcd_mvcc_db_total_size_in_bytes vs ..._in_use_bytes	Difference is fragmentation; large gaps mean defrag time.
etcd_network_peer_round_trip_time_seconds	Per-peer RTT histogram.
etcd_grpc_server_started_total{grpc_method=...} / ..._handled_total	Per-RPC volume + outcome.
etcd_lease_granted_total, etcd_lease_expired_total	Lease churn.
process_resident_memory_bytes, go_gc_duration_seconds	Standard Go runtime metrics.

Mixin alerts

contrib/mixin/mixin.libsonnet defines alerting rules that wrap the metrics above. The library is consumed via jsonnet-bundler and emits the alerts/dashboards at make time.

Adding a new metric

1. Declare a prometheus.NewCounterVec or similar in the package's metrics.go.
2. prometheus.MustRegister(...) in an init() (or the constructor).
3. Update tools/etcd-dump-metrics/'s expected output if the test asserts on the catalogue.
4. Add a test that exercises the metric path; metrics tests typically use prometheus.testutil.GatherAndCompare.

Cross-references

• systems/v3rpc — gRPC-level metrics.
• systems/wal — fsync metrics.
• deployment — operator dashboards.