Debugging

Pragmatic notes for diagnosing problems in a running etcd member or in the test suite.

Logs

etcd uses zap under go.uber.org/zap everywhere. Flags from server/embed/config.go:

• --logger=zap (default; only zap is supported in current versions)
• --log-level=debug|info|warn|error
• --log-outputs=stderr,/var/log/etcd.log
• --log-format=json|console

The leader records "applied" timing for every request; if a request takes longer than --warning-apply-duration (default 100 ms), you get a structured warning with the operation type, key, and trace span. The server-wide trace threshold is 100 * time.Millisecond (server/etcdserver/v3_server.go, traceThreshold).

Metrics and pprof

• :2379/metrics — Prometheus metrics (server/etcdserver/api/etcdhttp/metrics.go).
• :2379/debug/pprof/* — gated by --enable-pprof (server/etcdserver/api/etcdhttp/debug.go).

See how-to-monitor/ for what is exposed.

Health and version endpoints

• :2379/health returns 200 with a JSON body when the member is healthy. Implementation in server/etcdserver/api/etcdhttp/health.go. The endpoint also drives Kubernetes readiness probes.
• :2379/version returns the cluster + server version (server/etcdserver/api/etcdhttp/version.go).

Inspecting the bolt DB

etcdutl (etcdutl/) and the dev tool tools/etcd-dump-db/ both let you read the on-disk bolt file directly:

./bin/etcdutl snapshot status snap.db
./bin/etcd-dump-db iterate-bucket /var/lib/etcd/member/snap/db key

tools/etcd-dump-logs/ dumps WAL records with their term/index for forensic work.

Tracing

Set --experimental-enable-distributed-tracing plus the OTLP exporter flags (--experimental-distributed-tracing-address, ...-service-name, ...-sampling-rate-per-million) to ship spans to a collector. Code: server/embed/config_tracing.go, server/etcdserver/tracing.go.

Reproducing common failure modes

Scenario	How to reproduce
Apply lag warnings	Use the tests/robustness/ "slow apply" failpoint (tests/robustness/failpoint/).
Quota exceeded	--quota-backend-bytes=1048576 then PUT until alarm fires.
Snapshot install on slow follower	Use --snapshot-count=10 on a 3-node cluster, generate writes faster than the follower can keep up.
Watch resume after disconnect	tests/integration/clientv3/watch/v3_watch_test.go covers most edge cases.

Getting stack traces

kill -QUIT <etcd-pid>            # full Go stack to stderr
curl :2379/debug/pprof/goroutine?debug=2

"It hung. Why?"

Most production hangs are one of:

• Backpressure on the apply loop (maxGapBetweenApplyAndCommitIndex reached).
• A bbolt write blocked because a long read transaction is open.
• Lease expiration storm (server/lease/lessor.go).
• A peer transport stuck on a slow rafthttp connection.

Each has dedicated metrics in server/etcdserver/metrics.go, server/storage/backend/metrics.go, server/lease/metrics.go, and server/etcdserver/api/rafthttp/metrics.go.