neondatabase/neon

Layers

A layer is the pageserver's storage atom: an immutable record of a key range across an LSN range. Every byte of pageserver data lives inside a layer. The full description is on the Storage layers page; this page covers the conceptual model.

Two flavors

Flavor	What it stores
Image layer	A snapshot. For each key in `[key_start, key_end)`, the page image at exactly `lsn`.
Delta layer	WAL records. For each key in `[key_start, key_end)`, the WAL records that touched it in `(lsn_start, lsn_end]`.

Both are immutable. Neither is updated in place. Compaction creates new layers; garbage collection deletes old ones.

Reading a page

Given (key, requested_lsn):

   image
   layer  ──── most recent image at LSN ≤ requested_lsn ──┐
                                                          │
   delta layer covering (image_lsn, requested_lsn] ──────┐│
   delta layer covering ...                              ││
                                                         ▼▼
                                               WAL-redo subprocess
                                                         │
                                                         ▼
                                                    materialized page

The number of delta layers in the chain is the redo depth for that key. If it grows too large, compaction creates a new image layer to short-circuit the chain.

Naming

Layer files have self-describing names. For a delta layer covering keys [A, B) over LSNs (C, D]:

000000000000000000000000A1A2A3A4-000000000000000000000000B1B2B3B4__C1C2C3C4-D1D2D3D4

(Lowercase 36-hex-character key range, two underscores, lowercase 16-hex-character LSN range.) Image layers omit the lower bound of the LSN range and use a different separator. The pageserver doesn't need an external index to know what's on disk — it can rebuild its layer map by listing the timeline directory and parsing filenames.

Layer map

The in-memory LayerMap (pageserver/src/tenant/layer_map.rs) is an interval tree keyed by (key_range, lsn_range). It answers two queries:

Point search. "Which layer covers (key, lsn)?" — used on every GetPage@LSN.
Range search. "Which layers cover (key_range, lsn_range)?" — used by compaction, basebackup, and disk eviction.

Modifications to the layer map are atomic, transactional, and rare relative to lookups (the read-mostly access pattern shapes the data structure choice).

In-memory ↔ on-disk

There's a third (transient) layer kind that exists only in RAM:

In-memory / ephemeral layer (pageserver/src/tenant/ephemeral_file.rs). New WAL from the safekeeper accumulates here. When it grows past checkpoint_distance (default 256 MB), it's frozen and written out as a new L0 delta layer file.

So at any moment a timeline's storage is:

One in-memory layer (the head of the WAL tail).
Many L0 delta layers (the recent history).
Many L1 delta layers and image layers (the older, compacted history).

L0 vs L1

These describe shape, not generation:

L0 — wide-keys, narrow-LSN. Output of an in-memory layer flush. One L0 covers the whole key space.
L1 — narrow-keys, wide-LSN. Output of compaction.

A GetPage@LSN for a key first scans L0 layers (every L0 has all keys, so any of them might be relevant) and then narrows in on a specific L1.

Layers in remote storage

Each layer file is mirrored to remote object storage (S3/GCS/Azure) via pageserver/src/tenant/remote_timeline_client.rs. The directory layout in the bucket mirrors the local one but is also accompanied by an index_part.json manifest that lists every layer the timeline owns.

A layer can be:

local + remote — present on both. Normal state.
remote only — evicted from local disk to free space. Will be downloaded on demand.
local only — newly created, queued for upload.

The pageserver's deletion queue (pageserver/src/deletion_queue.rs) ensures layers are deleted from remote storage in a safe order — index_part.json is updated to no longer reference a layer before the object is deleted, so a crash mid-deletion can never lose data.

Sharded layers

When a tenant is sharded across N pageservers, each shard owns a strict subset of the key space. Layer files are partitioned by shard: shard 0 has its own complete set of layers, shard 1 has its own, and so on. There is no shared layer between shards. The shard a key belongs to is determined by ShardIdentity::get_shard_number(key) (libs/pageserver_api/src/shard.rs).

Compression and checksums

Layer files compress their blob payload (page images and WAL records) using LZ4 by default; configurable. Each block has a CRC checksum to detect bitrot. A layer with a checksum mismatch is evicted from local disk and re-downloaded from remote.

Why immutable

Several reasons:

Cache friendliness. A layer can be safely cached at any layer of the stack (page cache, OS cache, S3 cache) because no one ever rewrites it.
Concurrency. Multiple readers and the compaction writer never conflict on the same file.
Crash safety. A partial write of a temporary file is just discarded on restart.
Replicability. The same layer file uploaded to two regions has identical bytes; deduplication and verification are trivial.

The cost is write amplification — every change eventually flows through compaction and (sometimes) image regeneration. The compaction algorithm in pageserver/src/tenant/timeline/compaction.rs is the offset.