Storage and MVCC

pkg/storage/ is CockroachDB's local storage layer. It wraps Pebble (github.com/cockroachdb/pebble) with CRDB-specific MVCC encoding, intent/lock encoding, encryption-at-rest, and SST ingestion. Every replica's data — keys, values, intents, and the Raft log — lives in a per-store Pebble database.

Purpose

Offer a durable, sorted, transactional KV map with multi-version concurrency control. The package layers MVCC semantics, the lock-table encoding, snapshot iteration, range key semantics (for MVCCDeleteRange), and Raft-log writes on top of Pebble's atomic batches and SSTable ingest.

Directory layout

pkg/storage/
├── doc.go                      // README-style overview of MVCC layout
├── engine.go                   // storage.Engine interface (~95 KB)
├── pebble.go                   // Pebble implementation (~125 KB)
├── pebble_mvcc_scanner.go      // optimized MVCC scan (~70 KB)
├── pebble_iterator.go          // CRDB-specific iterator wrapper
├── mvcc.go                     // MVCC read/write paths (~340 KB)
├── mvcc_key.go                 // MVCC key encoding/decoding
├── mvcc_value.go               // MVCC value envelope
├── intent_interleaving_iter.go // legacy interleaved-intent iteration
├── lock_table_iterator.go      // separated lock-table iterator
├── min_version.go              // cluster-version ↔ Pebble format major
├── encryption.go               // EAR plumbing (open source side)
├── sst.go / sst_writer.go      // SST construction
├── external_sst_reader.go      // read shared/external SSTs
├── shared_storage.go           // disaggregated-storage entry
├── replicas_storage.go         // per-replica storage layout
├── disk/                       // disk monitoring
├── enginepb/                   // proto types: MVCCMetadata, etc.
├── fs/                         // VFS abstraction
├── pebbleiter/                 // pebble.Iterator wrapper
├── storageconfig/              // engine config types
├── mvcceval/                   // shared evaluator helpers
└── metamorphic/                // randomized metamorphic tests

Key abstractions

Type	File	Description
Engine	pkg/storage/engine.go	The main storage interface
Reader / Writer	pkg/storage/engine.go	Sub-interfaces for read/write contexts
Pebble	pkg/storage/pebble.go	Production engine
inMem	pkg/storage/in_mem.go	In-process Pebble for tests
Batch	pkg/storage/batch.go	Write batch with read-after-write semantics
MVCCScanOptions	pkg/storage/mvcc.go	Knobs for MVCC scans
SSTWriter	pkg/storage/sst_writer.go	Build a SST for ingestion
MultiCompactionScheduler	pkg/storage/multi_compaction_scheduler.go	Coordinate multi-store compactions

MVCC encoding

CockroachDB stores every value with a timestamp suffix in big-endian decreasing order. The on-disk layout for a key k looks like:

k                      → MVCCMetadata (only present if there is an unresolved intent)
k_<ts_n>               → MVCCValue (latest version)
k_<ts_{n-1}>           → MVCCValue
…
k_<ts_0>               → MVCCValue

A MVCCMetadata (proto in pkg/storage/enginepb/) only exists while there is an unresolved write intent. The "intent" version uses sequence numbers and may be rolled back via savepoints. After commit, the intent is resolved into a regular versioned key.

pkg/storage/doc.go is the canonical reference for the encoding and explains why metadata can be elided. The decreasing-timestamp encoding ensures that the most recent version sorts immediately after the metadata key, which Pebble's bloom filters and block cache exploit.

Write paths

Operation	File	Notes
MVCCPut	mvcc.go	Writes a new versioned value; updates metadata if intent.
MVCCBlindPut	mvcc.go	Bypasses meta lookup when caller knows there is no intent.
MVCCConditionalPut	mvcc.go	CAS on previous value.
MVCCInitPut	mvcc.go	Idempotent put.
MVCCDeleteRange	mvcc.go	Range tombstone using Pebble range keys.
MVCCResolveWriteIntent	mvcc.go	Commit/abort intent in place.
MVCCGarbageCollect	mvcc.go	Drop versions older than gc_threshold.
MVCCRefreshRange	mvcc.go	Verify no writes in [from, to] after t1.

Read paths

The hot read path is pkg/storage/pebble_mvcc_scanner.go (~70 KB). It implements MVCCGet, MVCCScan, and MVCCReverseScan over a Pebble iterator. The scanner inlines bounds checks, intent resolution, and limit/maxKeys bookkeeping for performance.

For incremental backups and CDC, pkg/storage/mvcc_incremental_iterator.go returns only versions in (start, end] with intent handling.

Lock table

pkg/storage/lock_table_iterator.go and pkg/storage/lock_table_key_scanner.go read the on-disk lock table — the persisted form of replicated locks. The "interleaving" iterator is a legacy path that walks intents inline with their MVCC keys; it is being replaced by the separated lock-table iterator in mixed-version-safe phases.

SSTables and ingestion

CockroachDB heavily uses Pebble's "ingest" interface. SSTs are built outside the LSM (typically by IMPORT, RESTORE, schema-change backfill, or replication ingestion) and atomically added to L6. Code:

• pkg/storage/sst.go and sst_writer.go build SSTs.
• pkg/storage/external_sst_reader.go reads SSTs that live in shared/cloud storage.
• pkg/storage/shared_storage.go is the entry for "disaggregated storage" — pushing the deepest LSM levels to S3-compatible object storage to reduce inter-AZ replication cost.

Encryption at rest

pkg/storage/encryption.go defines the engine-level interface; the actual implementation that uses customer-managed keys lives under pkg/ccl/storageccl/engineccl/ and is gated by the CSL. The OSS side allows opting into encryption with a static key for testing (auto_decrypt_fs.go).

Cluster-version coupling

pkg/storage/min_version.go ratchets Pebble's "format major version" alongside the CRDB cluster version. When a node observes a new active cluster version, Engine.SetMinVersion enables the corresponding Pebble format and writes a STORAGE_MIN_VERSION file outside the store so the next process start does not need to open the store to know what format to expect. doc.go explains the failure-recovery story when a crash interrupts the ratchet.

Replicas storage

pkg/storage/replicas_storage.go is the API used by kvserver to construct a replica's view of the engine — its raft state, applied state, range descriptors, and locks — without leaking Pebble specifics into kvserver code.

Tests

• MVCC history tests (mvcc_history_test.go, ~92 KB plus testdata) — datadriven scenarios for every MVCC operation in every order.
• Metamorphic tests (pkg/storage/metamorphic/) — randomly-generated workload that runs against multiple engines and asserts equivalence.
• Pebble fuzzer — wired into Pebble's own fuzz harness; some entry points live here.

Entry points for modification

• Add an MVCC operation: extend mvcc.go and pebble_mvcc_scanner.go; add datadriven cases in testdata/mvcc_histories/.
• Bump Pebble: bump go.mod, DEPS.bzl, possibly add a new format major version mapping in min_version.go, and gate behind a cluster version.
• New SST consumer: use SSTWriter and ingest via Engine.IngestExternalFiles or IngestRangeKeys.

Related pages

• KV server — the only direct caller.
• Cluster version & upgrade — Pebble format major versions are gated this way.
• features/disaggregated-storage — shared-storage details.