KV server

pkg/kv/kvserver/ is where ranges live. Every node opens one or more Stores, each backed by a Pebble engine, and each Store hosts many Replicas. A replica owns a Raft group, evaluates BatchRequests, applies committed entries to the MVCC layer, and runs the queues that keep the cluster healthy.

Purpose

The package implements the server side of the KV API: the on-the-wire BatchRequest enters here, is admitted, latched, locked, evaluated against MVCC, possibly proposed through Raft, and applied. It also owns range-level maintenance — splits, merges, replication, lease transfers, GC, raft-log truncation, snapshot send/receive, and time-series rollups.

High-level flow

graph TD
  In["BatchRequest"] --> Send["replica_send.go"]
  Send --> Latch["spanlatch.Manager"]
  Latch --> Concurrency["concurrency.Manager (lock table)"]
  Concurrency --> Eval["batcheval / replica_evaluate.go"]
  Eval --> Branch{readonly?}
  Branch -->|yes| Read["replica_read.go"]
  Branch -->|no| Propose["replica_proposal_buf.go"]
  Propose --> Raft["pkg/raft (Raft group)"]
  Raft --> Apply["replica_application_state_machine.go"]
  Apply --> MVCC["storage MVCC + Pebble"]
  Apply --> SideEffects["lease, split, merge, …"]

Directory layout

pkg/kv/kvserver/
├── replica*.go                 // ~50 files defining Replica
├── store*.go                   // Store, queues, gossip
├── batcheval/                  // request evaluation per command
├── concurrency/                // lock table, latch manager glue
├── spanlatch/                  // per-key in-memory latches
├── intentresolver/             // resolve abandoned intents
├── txnrecovery/                // recover prepared 2PC transactions
├── txnwait/                    // queue for pushers waiting on a txn
├── tscache/                    // timestamp cache (read tracking)
├── closedts/                   // closed timestamp publication
├── liveness/                   // node liveness records
├── storeliveness/              // store-liveness fabric
├── leases/                     // lease types (epoch / store-liveness)
├── allocator/                  // replica placement decisions
├── kvflowcontrol/              // admission + flow-control over Raft
├── kvadmission/                // KV admission control adapter
├── logstore/                   // raft log / write-ahead log over Pebble
├── raftentry/                  // entry cache
├── raftlog/                    // entry encoding
├── rangefeed/                  // server-side change-feed
├── rditer/                     // replicated-data iterators
├── reports/                    // replication-status reports
├── kvflowcontrol/              // flow-control v2
├── apply/                      // applied state machine helpers
├── protectedts/                // protected timestamps (server-side)
└── … many more

Key types

Type	File	Description
Replica	pkg/kv/kvserver/replica.go	Per-range state machine. Owns Raft, lock table, latches, MVCC, side effects. ~125 KB.
Store	pkg/kv/kvserver/store.go	Hosts many replicas. Owns the engine, queues, gossip, raft scheduler. ~180 KB.
Stores	pkg/kv/kvserver/stores.go	Multi-store collection on a node.
concurrency.Manager	pkg/kv/kvserver/concurrency/manager.go	Coordinates latches and the lock table.
spanlatch.Manager	pkg/kv/kvserver/spanlatch/manager.go	Per-key latches that serialize concurrent ops.
Allocator	pkg/kv/kvserver/allocator/allocatorimpl/	Placement decisions (add/remove/move replicas).
replicateQueue	pkg/kv/kvserver/replicate_queue.go	Drives allocator decisions.
splitQueue / mergeQueue	pkg/kv/kvserver/split_queue.go, merge_queue.go	Sizes ranges.
mvccGCQueue	pkg/kv/kvserver/mvcc_gc_queue.go	Garbage-collects old MVCC versions.
raftLogQueue / raftLogTruncator	pkg/kv/kvserver/raft_log*.go	Trims the raft log.
RangeFeed	pkg/kv/kvserver/rangefeed/	Server-side rangefeed processor.

Lifecycle of a write

1. Admission — replica_send.go enters; flow control (kvflowcontrol/) and admission control (kvadmission/) decide whether to throttle.
2. Latching — concurrency.Manager.SequenceReq acquires latches on the keys/spans the request touches and waits in the lock table for blocking transactions.
3. Evaluation — replica_evaluate.go and batcheval/ evaluate the request against an MVCC snapshot, producing a WriteBatch plus side-effect metadata.
4. Proposal — for writes, the result is proposed to Raft via replica_proposal_buf.go. The proposal includes a kvserverpb.RaftCommand.
5. Replication — Raft replicates the entry; pkg/raft decides when it can be considered committed.
6. Application — replica_application_state_machine.go decodes committed entries, applies the WriteBatch to Pebble, and runs side effects (split trigger, merge trigger, lease change, raft log truncation, GC, …).
7. Reply — the proposal's waiting goroutine is signaled; the response travels back up to the client.

For reads, the path skips Raft: replica_read.go evaluates the request against a Pebble snapshot at a chosen timestamp and returns directly.

Concurrency control

The lock table (pkg/kv/kvserver/concurrency/) holds three kinds of locks:

• Replicated locks — written to MVCC as intents.
• Unreplicated locks — in-memory only.
• Replicated unreplicated locks — used by the lock-table iterator.

A request's locking spans are declared up-front; concurrency.Manager.SequenceReq blocks until they are free, with deadlock detection through the txn wait queue (pkg/kv/kvserver/txnwait/). When a request encounters a write intent, it pushes the owning transaction (pkg/kv/kvserver/intentresolver/) — the push either bumps the pusher's read timestamp, aborts the pushee, or waits for the pushee to commit.

Leases and store liveness

A range's writes (and most reads) go to its leaseholder. CockroachDB has two lease types in active use:

• Epoch-based leases — backed by node liveness epochs in pkg/kv/kvserver/liveness/. Currently being phased out.
• Store-liveness leases — backed by the per-store fabric in pkg/kv/kvserver/storeliveness/ and integrated into Raft via pkg/raft/raftstoreliveness/. New deployments default to these.

Lease handling lives in pkg/kv/kvserver/leases/ and replica_range_lease.go.

Queues

Each Store runs ~10 queues (pkg/kv/kvserver/queue.go is the framework). Notable ones:

Queue	What it does
replicateQueue	Add/remove/move replicas via the Allocator
splitQueue	Split oversize ranges
mergeQueue	Merge undersized adjacent ranges
mvccGCQueue	Garbage-collect old MVCC versions
raftLogQueue	Trigger Raft-log truncation
raftSnapshotQueue	Send pending Raft snapshots
consistencyQueue	Periodically checksum each range across replicas
replicaGCQueue	Reap zombie replicas
tsMaintenanceQueue	Roll up time series
leaseQueue	Move leases to balance load

A Store rebalancer (store_rebalancer.go) and a multi-metric load-aware rebalancer (mma_store_rebalancer.go, mma_replica_store.go) move ranges based on QPS, CPU, and disk usage.

Rangefeed

A rangefeed is a streaming change-feed of a span: the server emits every committed value, plus checkpoints derived from closed timestamps. pkg/kv/kvserver/rangefeed/ implements the per-replica processor; pkg/kv/kvclient/rangefeed/ is the client side. Used by:

• Changefeed jobs (CDC).
• Span-config reconciliation (pkg/spanconfig/spanconfigkvsubscriber/).
• Protected-timestamp reconciliation.
• Cross-cluster replication.

Loss-of-quorum recovery

pkg/kv/kvserver/loqrecovery/ and pkg/cli/debug_recover_loss_of_quorum.go implement an offline recovery procedure for clusters that have lost a majority of replicas — either by manually marking the surviving replica as the new leader or by stripping out replicas that are gone.

Observability and admin

• metrics.go (~230 KB) — every store and replica metric.
• range_log.go — the system event log entries for splits/merges/replication changes.
• metric_rules.go — evaluation rules used by the DB Console.
• pkg/kv/kvserver/asim/ — a deterministic Allocator simulator.

Entry points for modification

• New KV command: declare it in pkg/kv/kvpb/api.proto, write its evaluator in pkg/kv/kvserver/batcheval/, register the spans it touches.
• New queue: implement replicaQueue and register on Store.
• New rangefeed event type: extend kvserverpb events and the rangefeed processor.
• New replication policy: extend pkg/kv/kvserver/allocator/.
• Lease changes: tread carefully — write a roachtest and a kvnemesis run.

Related pages

• Raft — the consensus library underneath.
• Storage / MVCC — what reads and writes ultimately touch.
• KV client — the API that fans out to this layer.
• Span config — how zone configs become per-range placement decisions.