turbo-persistence

The on-disk storage backend for turbo-tasks's persistent cache. An LSM-style key-value store tuned for Turbopack's access patterns.

Source: turbopack/crates/turbo-persistence/src/. Heavy files: db.rs (91 KB), static_sorted_file_builder.rs (69 KB), tests.rs (64 KB), static_sorted_file.rs (41 KB).

Why a custom store?

Turbopack's persistent cache has unusual properties:

• Massive write amplification — every recompile invalidates and rewrites cells.
• Read-heavy after warmup — once the cache is hot, dev rebuilds read most of it without writing.
• Append-only is a great fit — task results are immutable per-input; new versions add new keys.
• Memory-mapped reads are critical for cold starts — devs care about dev-server startup time.

Off-the-shelf KV stores either don't optimize for this shape (RocksDB writes are fine but reads underutilize mmap) or are too generic (sqlite). turbo-persistence is a hand-rolled LSM tree where:

• SST files are mmap-friendly and small (memory-residence per file is cheap).
• Compaction is manual, scheduled by the framework rather than running automatically.
• The format is bincode-based — no schema migrations between versions; just bump a version and reset the cache.

File layout

The on-disk store consists of:

• Meta files (.meta) — small JSON-like manifests of the SSTs that make up the current generation.
• SST files (.sst) — Static Sorted Tables — the immutable batches of key-value pairs.
• Compaction artifacts — created during compaction, replace older SSTs.

Components

db.rs (91 KB)

The top-level TurboPersistence type. Exposes:

• open() / create()
• read() / read_batch()
• write_batch()
• compact()
• close()

Owns the meta-file management, the SST cache, and the write-batch coordination.

write_batch.rs (25 KB)

A staging buffer for writes. Each batch:

1. Accumulates entries in memory.
2. On commit, writes them to a new SST.
3. Updates the meta to point at the new SST.

static_sorted_file.rs + static_sorted_file_builder.rs

The SST format. Each SST has:

• A header with version, entry count, key range.
• A sorted run of key-value entries.
• A bloom filter (sst_filter.rs) for fast "key not present" checks.

The builder is the writer; the static file is the reader.

compaction/

The compaction strategy. Periodically, multiple SSTs get merged into one larger SST to keep read amplification bounded. Manual scheduling lets Turbopack pick low-impact moments (idle dev server) for compaction.

meta_file.rs + meta_file_builder.rs

The meta files that point at the current generation of SSTs. Rotated on every commit.

merge_iter.rs

A k-way merge iterator over multiple SSTs, used both for reads and for compaction.

collector.rs + collector_entry.rs

The "write side" data structure that buffers writes until they're flushed to an SST.

Bytes / mmap

• arc_bytes.rs, rc_bytes.rs, shared_bytes.rs — refcounted byte slices for zero-copy reads.
• mmap_helper.rs — mmap wrapper.
• compression.rs — opt-in compression for value blobs.

parallel_scheduler.rs

A small executor that runs compaction in parallel without contending with the main turbo-tasks scheduler.

value_block_count_tracker.rs, value_buf.rs, lookup_entry.rs

Plumbing for reading values stored across multiple SST blocks.

key.rs, be.rs, constants.rs

Key encoding and constants.

bin/

Standalone tools for inspecting the store.

Tests

tests.rs (64 KB) is property-tested: random sequences of writes, reads, and compactions are exercised against the store with invariants checked at each step.

Integration with turbo-tasks

turbo-tasks-backend uses turbo-persistence to spill cells to disk:

• Each cell gets a key derived from its task input hash.
• Cell contents are stored as values.
• On startup, cells lazy-load from disk as tasks ask for them.
• On invalidation, the corresponding key is overwritten in the next batch.

The schema lives in turbopack/crates/turbo-tasks-backend/src/backend/storage_schema.rs — that's the layer that knows how to translate a task into a key.

Why opt-level = 1?

The root Cargo.toml overrides this crate's debug-build optimization level:

[profile.dev.package.turbo-persistence]
opt-level = 1

Reason: turbo-persistence dominates dev-server cold-start time. Even debug builds need it fast or pnpm dev becomes painful. The rest of the workspace stays at default debug (opt-level = 0) for compile speed.

Inspection

The turbo-persistence-tools crate (turbopack/crates/turbo-persistence-tools/) provides CLI tools for opening a store and dumping its contents.

Key source files

File	Purpose
turbopack/crates/turbo-persistence/src/lib.rs	Crate entry
turbopack/crates/turbo-persistence/src/db.rs	Top-level TurboPersistence
turbopack/crates/turbo-persistence/src/write_batch.rs	Write-batch staging
turbopack/crates/turbo-persistence/src/static_sorted_file.rs	SST reader
turbopack/crates/turbo-persistence/src/static_sorted_file_builder.rs	SST writer
turbopack/crates/turbo-persistence/src/compaction/	Compaction strategy
turbopack/crates/turbo-persistence/src/meta_file.rs	Generation metadata
turbopack/crates/turbo-persistence/src/merge_iter.rs	K-way merge
turbopack/crates/turbo-persistence/src/sst_filter.rs	Bloom filter
turbopack/crates/turbo-persistence-tools/	Inspection tools