Compaction

Compaction is the background process that turns the L0 layer backlog into L1 layers and (less frequently) regenerates image layers. Without it, the read path's redo chain would grow without bound.

What needs to happen

Recall from Storage layers:

• L0 layers cover the whole key space and a narrow LSN range. They pile up as fast as the in-memory layer flushes.
• L1 layers cover a narrow key range and the whole LSN range since the last image cut. To answer a GetPage@LSN you need to consult at most one or two L1 layers, not many L0s.
• Image layers are key-range snapshots at a specific LSN. They cap the depth of the WAL-redo chain.

Compaction is the conversion engine.

Phases

graph TD
    L0Pile[Many L0 deltas<br/>whole-key, narrow-LSN] -->|L0 → L1 compaction| L1[Few L1 deltas<br/>narrow-key, wide-LSN]
    L1 -->|hot key range with deep redo chain| Image[New image layer at chosen LSN]
    L1 -->|outside PITR window| GC[Garbage collected]

The pageserver runs three loosely-coupled jobs:

1. L0 compaction — when the number of L0 layers exceeds a threshold, pick a key-range slice and rebuild the affected layers as L1s. The output L1 covers the same keys but spans the LSN range of all the consumed L0s.
2. Image layer generation — when reading a key would require replaying many delta records, write a new image layer at a freshly chosen LSN. Subsequent reads of that key range short-circuit at the image.
3. Garbage collection — drop layer files entirely below the timeline's PITR cutoff. After GC, basebackup at LSNs before the cutoff is no longer possible.

L0 → L1

pageserver/src/tenant/timeline/compaction.rs (≈4.6k lines, the core algorithmic file) selects a target key range, opens the L0 layers and any L1 layers that overlap, and emits one or more new L1 layer files.

The algorithm is a classic LSM-style merging compaction with a few wrinkles:

• It honors the timeline's PITR retention, so it doesn't drop deltas that are still within the keep-window even if they would be redundant for the latest LSN.
• It re-uses the disk-btree builder from pageserver/src/tenant/disk_btree.rs to write the output layers' indexes.
• It schedules I/O cooperatively through the task_mgr to avoid starving the page service.

Image layer generation

When the chain of delta layers above a key region grows beyond a configurable threshold (image_layer_creation_threshold in pageserver/src/config.rs), the compaction job materializes a new image layer:

1. Pick a key range and an LSN (typically the timeline's last_record_lsn).
2. For each key in the range, ask the timeline for the materialized page at that LSN (this is itself a get_at_lsn call that goes through layer map and WAL-redo).
3. Stream the resulting (key, page) pairs into a new image layer.

Image generation is expensive — it's effectively a synchronous read of every page in the key range. The pageserver schedules it during low-load periods and limits the number of in-flight image generations per tenant.

Garbage collection

pageserver/src/tenant/gc_block.rs and the GC code paths in tenant/timeline.rs drop layers whose LSN range falls entirely below the timeline's PITR horizon. A layer is also kept if any descendant timeline (branch) needs it at its ancestor_lsn.

A child branch can pin its ancestor's layers indefinitely. To free them, a user can run cargo neon timeline detach_ancestor (pageserver/src/tenant/timeline/detach_ancestor.rs, ≈48 KB), which materializes the branch's data into its own layers and severs the dependency.

Tunables

In pageserver/src/config.rs:

Name	Meaning
checkpoint_distance	When the in-memory layer reaches this many bytes, freeze and write an L0.
compaction_threshold	How many L0 layers to allow before triggering L0→L1.
compaction_period	How often to wake up the compaction task.
gc_period	How often to wake up the GC task.
image_creation_threshold	How deep a redo chain to tolerate before generating an image.
pitr_interval	How far back in time the timeline keeps WAL accessible.

Most of these can be overridden per tenant. The storage controller can also override them centrally for fleet-wide changes.

Failure modes and fail-safes

• Mid-compaction crash. Compaction always writes new layers under a temp filename and renames into place atomically. If the pageserver crashes mid-compaction, the partial output is discarded on restart.
• Index corruption. Layer files include checksums; corrupted layers are evicted and re-downloaded from remote storage.
• Compaction backpressure. If compaction can't keep up with WAL ingest, L0 count grows; this both hurts read latency and prevents the in-memory layer from rotating. The pageserver_layer_count metric and L0 layer count alarms are the primary signals.

Sub-supporting crate

pageserver/compaction/ is a standalone crate with the algorithmic core (key partitioning helpers, scheduler, etc.). It's separated from pageserver so that the pagebench tool and unit tests can exercise it without spinning up a full pageserver.

Key source files

File	Purpose
pageserver/src/tenant/timeline/compaction.rs	The main compaction state machine.
pageserver/compaction/src/	Algorithmic helpers shared with tooling.
pageserver/src/tenant/gc_block.rs	Garbage collection guard.
pageserver/src/tenant/gc_result.rs	GC result reporting.
pageserver/src/tenant/timeline/detach_ancestor.rs	Branch detach (materializes a branch's data so GC of ancestor can proceed).
docs/pageserver-compaction.md	Long-form description in repo docs.

See also

• Storage layers — the on-disk format compaction is rewriting.
• Pageserver — the surrounding system.