Deployment

How real-world etcd clusters are deployed and operated. The repo intentionally ships only the building blocks; the etcd.io docs site is the canonical operational guide. This page indexes what is in-tree.

Reference

• Operating etcd — clustering, recovery, performance, security, monitoring.
• Documentation/postmortems/ — the project's incident retrospectives.
• security/ — security release process, PGP keys, audits (SECURITY_AUDIT.pdf, FUZZING_AUDIT_2022.PDF).

In-tree deployment artifacts

Path	What it is
Procfile	Local 3-node cluster definition for goreman
etcd.conf.yml.sample	Documented YAML config sample
Dockerfile	Multi-stage Docker build of the etcd binaries
scripts/build-docker.sh	Helpers used by the release pipeline
contrib/systemd/	Sample systemd unit files
contrib/mixin/	Grafana dashboards + Prometheus alerts (Jsonnet)
contrib/lock/	Reference distributed-lock client
contrib/raftexample/	A self-contained example app embedding Raft directly
tools/local-tester/	Tiny multi-node deterministic-fault test harness

The Grafana mixin

contrib/mixin/ is a monitoring-mixin package: a Jsonnet library that emits Prometheus alerts and Grafana dashboards. Operators import it via jsonnetfile.json to get a ready-to-go monitoring story.

Provided dashboards:

• etcd-cluster.json, etcd.json — cluster overview, KV throughput, leader changes, disk fsync latency.
• etcd-on-kubernetes.json — Kubernetes-tailored variant (selectors for the kube-system pods).

Alerts cover: leader changes / no leader, member unhealthy, high latency, high lease churn, fsync slowness, NOSPACE alarm.

Hardware sizing

The README and the docs site cite empirical numbers; for code-level context:

• WAL fsync is the dominant on-disk concern — every committed entry triggers a sync. SSDs strongly recommended.
• bbolt's mmap region grows as the DB does; --quota-backend-bytes should be set well below RAM size.
• Network: peer traffic (default :2380) tolerates single-digit ms RTT; the Raft heartbeat default is 100 ms but should be tuned for cross-AZ deployments.

Common topologies

Topology	Notes
3-node cluster	The standard deployment; tolerates 1 failure
5-node cluster	Used for larger Kubernetes deployments; tolerates 2 failures
3-node + learner	Recommended when joining a new member to an existing cluster
1-node embedded	embed.Etcd inside another Go binary; loses fault-tolerance
etcd gateway start in front	Single endpoint for client connections
etcd grpc-proxy start in front	Caching proxy for many clients

Backup and restore

etcdctl snapshot save produces a .db file consumable by etcdutl snapshot restore. See features/snapshots. The official ops docs detail point-in-time restore procedures, including the rare but documented case of recovering from quorum loss.

Upgrades and downgrades

• Rolling upgrade is the supported pattern: replace one member at a time, wait for the cluster to be healthy, repeat.
• Downgrade between adjacent minor versions is supported via etcdctl downgrade enable <version> and etcdctl downgrade cancel. Implementation: server/etcdserver/version/.
• etcdutl migrate operates on a stopped member's data dir — the offline counterpart.

CI/CD related

• .github/workflows/verify-released-assets.yaml smoke-tests the binaries published to GitHub Releases.
• scripts/release.sh orchestrates the cross-build, sign, and Docker-image-push pipeline.
• The signed checksums and SBOM are produced from bill-of-materials.json and verified by make verify-bom.