Pageserver

The pageserver is Neon's storage engine. It receives WAL from safekeepers, slices it into per-key data structures, persists those slices as immutable layer files on local disk and in cloud object storage, and answers GetPage@LSN requests from compute nodes by reconstructing page versions through a Postgres WAL-redo subprocess.

It's the largest crate in the repo (≈700 source files) and the central state machine of the storage stack. This page is the entry point; deeper topics have their own sub-pages.

Purpose

A pageserver process owns:

• Many tenants (one per Neon customer; multitenancy is documented in docs/multitenancy.md).
• Each tenant owns many timelines (branches).
• Each timeline owns a set of layer files that, together, contain enough information to reconstruct any page version at any LSN within the tenant's PITR window.

The pageserver's contract:

1. Accept WAL from one safekeeper per timeline (the WAL receiver).
2. Persist that WAL durably (locally first, then to remote storage).
3. Reconstruct any page at any in-window LSN on demand (the page service + WAL-redo).
4. Stream a "basebackup" tarball to a starting compute node so it can boot.

It also offers an HTTP management API used by the storage controller for tenant placement, attachment, and migration.

Directory layout

pageserver/
├── src/
│   ├── lib.rs                  # crate root, panic handler installation
│   ├── bin/                    # binaries (pageserver, dump_layerfile_to_stdout, …)
│   ├── config.rs               # config loading; ~36 KB
│   ├── tenant.rs               # central state machine; ~13k lines
│   ├── tenant/
│   │   ├── timeline.rs         # per-timeline state; ~8k lines
│   │   ├── timeline/
│   │   │   ├── compaction.rs   # L0→L1 compaction, image generation; ~4.5k lines
│   │   │   ├── walreceiver.rs  # ingest from safekeeper
│   │   │   ├── delete.rs       # timeline deletion
│   │   │   ├── detach_ancestor.rs  # branch promotion
│   │   │   └── eviction_task.rs    # disk-usage-driven layer eviction
│   │   ├── storage_layer.rs    # layer trait + image / delta layer impls
│   │   ├── layer_map.rs        # in-memory index of layers
│   │   ├── ephemeral_file.rs   # in-memory layer backing
│   │   ├── remote_timeline_client.rs  # talks to S3/GCS/Azure; ~3.3k lines
│   │   └── mgr.rs              # tenant manager
│   ├── page_service.rs         # libpq GetPage@LSN server; ~4.7k lines
│   ├── walredo.rs              # WAL-redo subprocess management
│   ├── walingest.rs            # decode + apply WAL to in-memory layer
│   ├── basebackup.rs           # build the bootstrap tarball
│   ├── basebackup_cache.rs     # cache of recent basebackups
│   ├── pgdatadir_mapping.rs    # map (rel, blknum) → key; ~3.5k lines
│   ├── http/                   # axum-based HTTP management API
│   ├── consumption_metrics.rs  # billing telemetry
│   ├── deletion_queue.rs       # multi-step S3 deletion
│   ├── disk_usage_eviction_task.rs  # global disk pressure handler
│   ├── virtual_file.rs         # io_uring-backed file abstraction
│   └── metrics.rs              # ~4.7k lines of Prometheus metrics
├── client/                     # libpq-based pageserver client (used by tests, neon_local)
├── client_grpc/                # gRPC client (newer)
├── compaction/                 # algorithmic layer-compaction support crate
├── ctl/                        # `pagectl` offline diagnostic CLI
├── page_api/                   # gRPC page-service definition
├── pagebench/                  # benchmark binary
└── benches/                    # criterion benches

How it works

graph TB
    subgraph Pageserver
        WR[WalReceiver<br/>walreceiver.rs] -->|decoded records| Mem[Ephemeral / In-memory layer<br/>ephemeral_file.rs]
        Mem -->|checkpoint, every<br/>checkpoint_distance| L0[L0 delta layer file]
        L0 -->|periodic compaction| L1[L1 delta + image layers]
        L0 & L1 -->|upload| Remote[(Remote storage<br/>remote_timeline_client.rs)]
        Remote -->|on-demand download| L1
        PS[Page service<br/>page_service.rs] -->|read base img + deltas| L1
        PS -->|missing image?<br/>replay deltas| Redo[WAL redo<br/>walredo.rs]
        Redo -->|pipe| RedoProc[(postgres --wal-redo)]
        Mgr[Tenant manager<br/>mgr.rs] -->|attach/detach| Tenant[Tenant<br/>tenant.rs]
        Tenant -->|owns N| TL[Timeline<br/>timeline.rs]
    end

    SK[Safekeeper] -->|libpq replication| WR
    Compute[Compute] -->|GetPage@LSN| PS
    Compute -->|basebackup| BB[basebackup.rs]
    Storcon[Storage Controller] -->|HTTP /v1/...| Mgr

A request lifecycle (read path):

1. Compute opens a libpq (or gRPC) connection to the page service.
2. It issues GetPage@LSN(tenant_id, timeline_id, key, lsn).
3. page_service.rs looks up the timeline, calls Timeline::get_at_lsn().
4. The layer map (layer_map.rs) finds the most recent image layer ≤ lsn containing key, and the delta layers covering (image_lsn, lsn].
5. If the needed layers aren't on disk, they are downloaded from remote storage (remote_timeline_client.rs).
6. The base image and the WAL records are passed to the WAL-redo manager (walredo.rs), which feeds them to a long-lived postgres --wal-redo subprocess via a pipe.
7. The redo process returns the materialized page; the page service serializes it back over the wire.

The write path:

1. The WAL receiver opens libpq replication to the safekeeper assigned to this timeline.
2. Each WAL record is decoded by wal_decoder (libs/wal_decoder/) into one or more (key, lsn, value) tuples.
3. These are appended to the in-memory layer (ephemeral_file.rs).
4. When the in-memory layer grows past checkpoint_distance, it's frozen, sorted, and written to a new L0 delta layer file.
5. A background task uploads the new file to remote storage and updates the index_part.json manifest.
6. Compaction periodically reshuffles L0 → L1 and produces image layers for hot key ranges.

Key abstractions

Type	File	Purpose
Tenant	pageserver/src/tenant.rs	Top-level per-tenant state, owns the WAL-redo process, the timeline map, and per-tenant config.
Timeline	pageserver/src/tenant/timeline.rs	Per-timeline state, layer map, last/disk/remote_consistent_lsn.
Layer	pageserver/src/tenant/storage_layer.rs	Trait implemented by image and delta layers.
LayerMap	pageserver/src/tenant/layer_map.rs	Indexed map of layers by key range × LSN range.
RemoteTimelineClient	pageserver/src/tenant/remote_timeline_client.rs	Uploads layers, manages index_part.json, downloads on demand.
WalRedoManager	pageserver/src/walredo.rs	Spawns and talks to the postgres --wal-redo subprocess.
PagestreamProtocolHandler	pageserver/src/page_service.rs	Implements the libpq replication protocol surface for GetPage@LSN.
TenantManager (mgr)	pageserver/src/tenant/mgr.rs	Owns the set of tenants, handles attach / detach / shard splits.

Sub-pages

• Storage layers — image and delta layers, layer map, on-disk format.
• WAL redo — how the pageserver runs Postgres in --wal-redo mode.
• Page service — the GetPage@LSN protocol over libpq and gRPC.
• Compaction — L0 → L1 reshuffling, image layer generation, garbage collection.

Integration points

• Receives WAL from: safekeepers (libpq replication, walreceiver.rs).
• Talks to: storage broker (subscribes to LSN updates, pageserver/src/controller_upcall_client.rs), storage controller (HTTP API for attach/detach), remote storage (libs/remote_storage).
• Serves: compute nodes (GetPage@LSN, basebackup), HTTP management API.
• Spawns: one postgres --wal-redo subprocess per tenant (pgxn/neon_walredo provides the postgres-side support).

Entry points for modification

• To change how WAL is decoded into per-page records, edit pageserver/src/walingest.rs and libs/wal_decoder/.
• To change layer file format, the most invasive change in this codebase: start at pageserver/src/tenant/storage_layer.rs and the image_layer/ and delta_layer/ submodules.
• To add an HTTP route or change request validation, see pageserver/src/http/routes.rs (≈4.3k lines).
• To change attach/detach or sharding behavior, work in pageserver/src/tenant/mgr.rs and pageserver/src/tenant.rs.

Useful docs in-repo

• docs/pageserver.md — high-level summary
• docs/pageserver-services.md — service breakdown (older but still accurate)
• docs/pageserver-storage.md — long-form storage description
• docs/pageserver-thread-mgmt.md — task management model
• docs/pageserver-walredo.md — WAL-redo subprocess details
• docs/pageserver-compaction.md — compaction algorithm