Pitfalls and danger zones

Places in the codebase where it is easy to introduce subtle correctness or performance bugs.

1. Apply must be deterministic

Every member runs apply/uberApplier against the same committed entry sequence. If apply ever does anything that varies across members — wall-clock-dependent, RNG-dependent, environment-dependent, or "skip on failure" — the cluster diverges.

Specific anti-patterns:

• Reading time.Now() and persisting it.
• Using os.Hostname() or environment variables in apply.
• Skipping a step on a per-member error (must be deterministic — either propose a follow-up or panic).
• Conditional logic on cluster version that only some members evaluate the same way.

The 3.5 inconsistency incident traced back to one of these.

2. Auth checks need to consider every path

The recent CVE-class fixes (70a2b4871, 4bc674b79, d97dfbc3b, c5893b5e6) all turned out to be apply paths that bypassed the gRPC-level auth interceptor:

• Put with prev_kv=true reading the previous value.
• Put with a lease the user doesn't own.
• Range reads via prev_kv in nested txns.

Whenever you add a code path that touches mvccpb.KeyValue on behalf of a user, run it past server/auth/range_perm_cache.go before returning data.

3. Long read transactions stall bbolt

A long-running read txn keeps the freelist pinned, which prevents the writer from reclaiming space. Symptoms: slowly-growing DB size, increasing fsync latency, intermittent NOSPACE alarms despite logical-size being small.

Hot spots to watch:

• server/storage/mvcc/watchable_store_test.go showed the original race.
• server/etcdserver/api/v3rpc/maintenance.go::Snapshot deliberately uses a streamed snapshot to avoid this.

4. Watch fragmentation and ordering

A WatchResponse may arrive in fragments (fragment=true). Clients must reassemble before delivery. The server's pump in server/etcdserver/api/v3rpc/watch.go is full of careful ordering: events under one revision must be delivered together; cancellation must not race with newly-published events.

5. Lease attach + put + revoke

A Put that attaches to an inactive lease must atomically fail; if it succeeded, you'd have orphan keys. The apply path computes the lease state inside the same transaction (server/etcdserver/txn/).

6. Conf changes are special

raftpb.EntryConfChange entries don't go through apply.Apply — they go through EtcdServer.applyConfChange. Any code path that reasons about "every applied entry" must handle both.

7. WAL truncation and Repair

If a WAL segment is partially written at crash time, wal.Repair truncates the trailing record. Looks innocuous, but has historically been the source of subtle bugs — never write WAL records that are not self-consistent under truncation.

8. Large ranges and pagination

Range with limit=0 returns everything in the requested range. Operators sometimes scan the whole keyspace and OOM the server. Production code should always paginate; the server now rejects ranges above --max-request-bytes worth of response.

9. gRPC stream close vs cancel

A Watch stream client may close their end (graceful) or have it dropped (sudden). The server must release watcher resources in both cases. The pattern in serverWatchStream.recvLoop is the canonical reference.

10. Bbolt mmap regions and Linux memory accounting

bbolt mmaps the entire DB; the kernel double-accounts the pages in some cgroup memory views. Operators occasionally see "etcd memory growth" that is purely page cache, then over-react and lower limits. The metric to trust is process_resident_memory_bytes — not what kubectl top shows, which often includes mmap pages.

11. Slow defrag

Defragment rewrites the entire DB and holds the global write lock during the rename. Run it offline (etcdutl defrag) when possible; if you must run it online, do it on followers first and use MoveLeader to avoid blocking client traffic on the leader.

12. Tests that touch real network

A few legacy tests bind to real ports and can be flaky under contention. The framework in tests/framework/integration/cluster.go provides ephemeral ports; new tests should always use it instead of hand-rolled net.Listen.