Apply pipeline

Source: server/etcdserver/apply/.

Purpose

After Raft commits an entry, the apply pipeline turns that entry into mutations against MVCC, leases, auth, and cluster membership. It is the "deterministic state machine" half of every Raft system. Getting apply wrong means corruption, divergence, or a lost update — so the package is full of invariants and explicit version gating.

Directory layout

server/etcdserver/apply/
├── apply.go            # toApply dispatch
├── interface.go        # apply Interface
├── uber_applier.go     # The "uberApplier" that wraps everything
├── backend.go          # Direct backend operations (Range, Put, DeleteRange)
├── auth.go             # Auth-related apply paths
├── corrupt.go          # Hash + corruption guard
├── capped.go           # Apply-rate guard
├── quota.go            # Backend-quota enforcement
└── metrics.go

Key abstractions

Symbol	File	Description
Interface	server/etcdserver/apply/interface.go	The apply contract — Range, Put, DeleteRange, Txn, Compaction, LeaseGrant, ..., AuthEnable, ..., Alarm, Cluster*, DowngradeInfoSet
applierV3 (uberApplier)	server/etcdserver/apply/uber_applier.go	Composition: applierV3backend → applierV3Capped → applierV3Corrupt → applierV3quota → ...
applierV3backend	server/etcdserver/apply/backend.go	Lowest-level apply — talks directly to mvcc/lease/auth
corruptGuard	server/etcdserver/apply/corrupt.go	Stops apply if peers' hashes disagree

How it works

graph TD
    raft[raftNode.toApply] --> srv[EtcdServer.applyAll]
    srv -->|EntryNormal| dispatch[apply.Apply]
    srv -->|EntryConfChange| confchange[applyConfChange]
    dispatch --> uber[uberApplier]
    uber --> capped[applierV3Capped]
    capped --> corrupt[applierV3Corrupt]
    corrupt --> quota[applierV3quota]
    quota --> auth[applierV3auth]
    auth --> base[applierV3backend]
    base --> mvcc[(mvcc.Store)]
    base --> lease[(lease.Lessor)]
    base --> authstore[(auth.AuthStore)]

Each layer adds one cross-cutting concern:

• capped — refuses writes if an alarm is active (NOSPACE or CORRUPT).
• corrupt — checks the peer hash agreement before applying.
• quota — refuses puts/txns above --quota-backend-bytes.
• auth — applies committed auth mutations and invalidates the permission cache.
• backend — performs the actual mvcc/lease writes.

The composition is built in uber_applier.go::newUberApplier.

Apply / commit gap

EtcdServer.applyAll reads toApply bundles in order; if it falls behind, the gap between Raft's committedIndex and the locally-applied index grows. The server backpressures new proposals when the gap exceeds maxGapBetweenApplyAndCommitIndex = 5000 (v3_server.go). Metrics etcd_server_apply_duration_seconds and etcd_disk_backend_commit_duration_seconds are the right places to look when this happens.

Conf change

Configuration changes follow a parallel path: EtcdServer.applyConfChange updates RaftCluster, reconfigures rafthttp, and (for promotion) flips a member's IsLearner flag.

Integration points

• Raft — driven by raftNode in etcdserver/raft.go.
• MVCC, Lease, Auth — the layered applier delegates to them.
• Alarm, corrupt, version — gates surfaced via separate apply files (v3alarm/, corrupt.go, etcdserver/version/).
• Tracing — every applied request can carry a traceutil.Trace that the server logs on slow apply.

Entry points for modification

• New gRPC method that mutates state → add to Interface and implement in applierV3backend (and any layer that needs to gate it).
• New cross-cutting guard → write a layer like corrupt.go and slot it into newUberApplier.
• New apply-time metric → apply/metrics.go.

Caveats

• Determinism is non-negotiable — every member must apply the same entry to the same effect. Anything time-of-day, random, or environment-dependent in apply has historically been a bug.
• Test coverage — uber_applier_test.go, auth_test.go, plus extensive integration tests under tests/integration/ and tests/robustness/.