Raft

pkg/raft/ is CockroachDB's fork of etcd/raft. Each range's Replica owns a RawNode and uses Raft to replicate its log. The fork retains the original API surface but adds CRDB-specific extensions: store-liveness-driven leadership, term cache, learner replicas, and metric integration.

Purpose

Provide a deterministic, replicated log abstraction. The application (here: Replica) hands committed entries one at a time to a state machine; Raft guarantees those entries are agreed-upon on a quorum of replicas before declaring them committed. This package is intentionally I/O-free — it produces "Ready" structs that callers persist and send.

Files

pkg/raft/
├── doc.go              // module-level documentation
├── design.md           // CRDB-specific design notes
├── raft.go             // ~115 KB — the protocol implementation
├── rawnode.go          // RawNode wrapper used by Replica
├── ready.go            // batched output (entries to persist + messages to send)
├── log.go / log_unstable.go  // in-memory log
├── storage.go          // pluggable persistent storage interface
├── term_cache.go       // CRDB extension: cache terms by index
├── status.go           // status struct used in metrics
├── types.go            // protocol types
├── confchange/         // joint-consensus configuration changes
├── quorum/             // quorum types (majority, joint majority)
├── tracker/            // per-peer progress tracking
├── raftpb/             // wire protocol types
├── raftstoreliveness/  // CRDB extension: leader leases backed by store liveness
├── raftlogger/         // logger plumbing
├── rafttest/           // datadriven test harness
└── testdata/

Key types

Type	File	Description
RawNode	pkg/raft/rawnode.go	The thin façade used by callers. Exposes Tick, Step, Ready, Advance.
raft	pkg/raft/raft.go	The actual protocol state machine.
ProgressTracker	pkg/raft/tracker/tracker.go	Per-peer match/next index, learner state, in-flight tracking.
Storage	pkg/raft/storage.go	Interface implemented by pkg/kv/kvserver/logstore/ and the in-memory test storage.
Ready	pkg/raft/ready.go	Output of one tick: entries to persist, snapshots to apply, messages to send, hard state to record.
StoreLiveness	pkg/raft/raftstoreliveness/	Adapter that exposes per-store liveness epochs to Raft.

How it integrates with kvserver

graph LR
  Replica["Replica (pkg/kv/kvserver)"] -->|Tick / Step / Propose| RawNode
  RawNode --> Raft["raft state machine"]
  Raft --> Ready
  Ready -->|entries| LogStore["logstore (Pebble WAL)"]
  Ready -->|messages| Transport["raft_transport.go"]
  Transport -->|gRPC stream| Peer["Peer Replica"]
  Replica -->|Apply| StateMachine["replica_application_state_machine.go"]

Replica.handleRaftReady (pkg/kv/kvserver/replica_raft.go) drives the loop: it ticks the RawNode, receives a Ready, persists entries to Pebble via the log writer, sends outgoing messages over the raft transport, and applies committed entries to the state machine. The transport is a long-running gRPC stream per peer (pkg/kv/kvserver/raft_transport.go) with admission control hooks (flow_control_raft_transport.go).

Store-liveness extension

The fork's biggest divergence is raftstoreliveness/. When enabled, leadership is anchored to a store-liveness epoch — the leader has the lease only as long as a quorum of peers fences for it. This replaces epoch-based leases with a fabric that does not require a single liveness range, eliminating a global hot spot. pkg/raft/raft.go consults the StoreLiveness to decide whether it can acquire leadership and whether peers are still vouching for the current leader.

Term cache

CockroachDB's Raft typically holds entries that span many terms. Mapping an index to a term used to be expensive (linear scan of the in-memory log). pkg/raft/term_cache.go is a small data structure that answers Term(index) → term in O(log n) by caching (firstIndex, term) pairs.

Configuration changes

pkg/raft/confchange/ implements joint-consensus configuration changes used to add/remove voters and learners. The CRDB schema-change-style usage:

1. Add a new replica as a LEARNER — receives the log but does not vote.
2. Once the learner has caught up via a snapshot, promote it to a VOTER.
3. The old voter is demoted to LEARNER then removed.

This lets replica relocation tolerate slow followers without reducing quorum size.

Snapshots

When a follower falls too far behind (Raft log truncation outpaces it), the leader sends a Raft snapshot. CockroachDB constructs the snapshot from MVCC by streaming SSTs (pkg/kv/kvserver/store_snapshot.go, kv_snapshot_strategy.go). The receiving side ingests them in snapshot_apply_prepare.go before re-creating the replica state.

Tests

pkg/raft/rafttest/ runs each Raft scenario as a datadriven test. The fork preserves the original etcd test corpus and adds CRDB-specific cases (leader leases, learners, store liveness). The full Raft paper test suite — pkg/raft/raft_paper_test.go — verifies that the implementation matches the figures in the Raft paper.

Entry points for modification

• Adjust election timing / heartbeat: tunables on Config in raft.go.
• New leader-side decision (e.g., quiescence): pkg/raft/raft.go::stepLeader.
• New Storage consumer: implement the Storage interface to plug a different log.
• Add a metric: register in pkg/raft/metrics.go and pkg/kv/kvserver/raft_transport_metrics.go.

Related pages

• KV server — the application layer above Raft.
• Storage / MVCC — where Raft entries and applied state both land.
• features/store-liveness — the store-liveness fabric.