neondatabase/neon

Safekeeper

The safekeeper is Neon's WAL service. A quorum of safekeepers durably accepts WAL records from a compute node, persists them, and forwards them to a pageserver. The protocol is a Paxos variant tuned for ordered append; the consensus model is documented (and machine-checked) in docs/safekeeper-protocol.md and the TLA+ specs under safekeeper/spec/.

Purpose

Each timeline (branch) of each tenant has its own quorum of safekeepers — typically three. The compute node runs a walproposer library inside the neon Postgres extension; the proposer streams WAL records to all members of the quorum. Once a majority acknowledge, the LSN is committed and the compute is free to consider it durable. The pageserver later pulls already-committed WAL from any safekeeper.

In Postgres terms:

The compute is the WAL proposer (also called the "primary").
The safekeeper is the WAL acceptor.
The pageserver is the WAL receiver.

The naming is intentional — the protocol borrows directly from Paxos vocabulary.

Directory layout

safekeeper/
├── Cargo.toml
├── src/
│   ├── bin/safekeeper.rs        # binary entry point
│   ├── lib.rs                   # library API
│   ├── safekeeper.rs            # core consensus state machine; ~67 KB
│   ├── timeline.rs              # per-timeline state; ~49 KB
│   ├── timeline_manager.rs      # owns the set of timelines
│   ├── timelines_global_map.rs  # hash-map index of timelines
│   ├── handler.rs               # libpq replication-protocol handler
│   ├── receive_wal.rs           # accept WAL from compute
│   ├── send_wal.rs              # forward WAL to pageserver / replicas
│   ├── send_interpreted_wal.rs  # newer interpreted-WAL fan-out
│   ├── recovery.rs              # cross-safekeeper catch-up after restart
│   ├── pull_timeline.rs         # bootstrap from another safekeeper
│   ├── copy_timeline.rs
│   ├── wal_storage.rs           # disk-backed WAL ring
│   ├── wal_backup.rs            # upload WAL segments to remote storage
│   ├── wal_backup_partial.rs
│   ├── control_file.rs          # persistent state header
│   ├── control_file_upgrade.rs  # schema migrations
│   ├── broker.rs                # storage-broker integration (LSN pub/sub)
│   ├── http/                    # axum-based management API
│   ├── metrics.rs
│   └── timeline_eviction.rs     # idle-timeline eviction
├── client/                      # rust client for the management API
├── benches/
├── tests/                       # integration + simulation tests
└── spec/                        # TLA+ specification of the consensus protocol

Protocol at a glance

sequenceDiagram
    participant C as Compute (walproposer)
    participant SK1 as Safekeeper 1
    participant SK2 as Safekeeper 2
    participant SK3 as Safekeeper 3
    participant PS as Pageserver

    Note over C,SK3: Election: walproposer collects voted_for/term from quorum
    C->>SK1: ProposerGreeting(term)
    C->>SK2: ProposerGreeting(term)
    C->>SK3: ProposerGreeting(term)
    SK1-->>C: AcceptorGreeting(term, flush_lsn)
    SK2-->>C: AcceptorGreeting(term, flush_lsn)
    SK3-->>C: AcceptorGreeting(term, flush_lsn)
    C->>C: pick highest LSN as starting point
    Note over C,SK3: Steady state: append + ack
    loop on each WAL record
        C->>SK1: AppendRequest(records, lsn)
        C->>SK2: AppendRequest(records, lsn)
        C->>SK3: AppendRequest(records, lsn)
        SK1-->>C: AppendResponse(flush_lsn)
        SK2-->>C: AppendResponse(flush_lsn)
        Note right of C: Quorum reached → record committed
    end
    PS->>SK2: replication start at LSN
    SK2-->>PS: stream WAL up to commit_lsn

The compute drives the protocol; safekeepers are passive acceptors. The starring guarantee is no committed record is ever lost: a record is committed only after a majority of safekeepers have flushed it to disk.

Key abstractions

Type / module	File	Purpose
`SafeKeeper`	`safekeeper/src/safekeeper.rs`	The consensus state machine. Implements `process_msg()` for each protocol message.
`Timeline`	`safekeeper/src/timeline.rs`	Per-timeline state: WAL storage, current term, commit/flush LSNs.
`WalStorage`	`safekeeper/src/wal_storage.rs`	The on-disk WAL ring buffer. Postgres-format WAL segment files.
`WalBackup`	`safekeeper/src/wal_backup.rs`	Periodically uploads completed segments to S3/GCS/Azure for archival.
`Broker`	`safekeeper/src/broker.rs`	Publishes `flush_lsn` to the storage broker so pageservers know where to pull from.
`ProposerHandler`	`safekeeper/src/receive_wal.rs`	Server side of the WAL-receive libpq protocol.
`WalSenderHandler`	`safekeeper/src/send_wal.rs`	Server side of the WAL-send libpq protocol used by pageservers.
`TimelineManager`	`safekeeper/src/timeline_manager.rs`	Schedules background work for one timeline (backup, eviction, recovery).

Persistence

Each safekeeper keeps per-timeline state on disk:

.neon/safekeepers/sk1/
├── safekeeper.id            # this safekeeper's stable id
└── <tenant_id>/
    └── <timeline_id>/
        ├── safekeeper.control   # binary control file (term, votes, LSNs)
        └── 000000010000000000000001  # WAL segments (Postgres format)

The safekeeper.control file is a binary header maintained by safekeeper/src/control_file.rs. Every WAL flush updates flush_lsn; every protocol message that changes the term or voted_for updates the control file before responding. Schema migrations of this file go through control_file_upgrade.rs.

WAL segments are full-sized (16 MB by default) Postgres WAL files. This means tools like pg_waldump work directly on a safekeeper's segment files, which has been useful for offline analysis many times.

Backup to remote storage

Once a WAL segment is fully written and committed, wal_backup.rs uploads it to remote storage via libs/remote_storage. This serves two purposes:

Disaster recovery. A safekeeper that loses its disk can rejoin the quorum by pulling segments back from S3.
Archival. Segments older than the safekeeper's retention can be trimmed locally and the cluster still has them in cold storage.

Partial (in-progress) segments are also uploaded periodically by wal_backup_partial.rs, with the trade-off being that a partial segment cannot serve replication until the next safekeeper writes the missing tail.

Recovery and pull

When a safekeeper starts up missing recent WAL — e.g. after disk loss or replacement — it can pull_timeline from another safekeeper that already has the data (safekeeper/src/pull_timeline.rs). The pull is incremental: only segments past the puller's flush_lsn need to be transferred.

recovery.rs implements safekeeper-side catch-up logic. It runs after restart and after term changes to make sure the state is consistent with the committed history.

Sending to the pageserver

The pageserver subscribes to a safekeeper using ordinary libpq physical replication (safekeeper/src/send_wal.rs). The connection is started at the pageserver's last disk_consistent_lsn. The safekeeper streams WAL records as fast as the pageserver can consume them.

A newer interpreted WAL sender (send_interpreted_wal.rs, ~47 KB) decodes WAL records on the safekeeper side and sends pre-decoded (key, lsn, value) tuples to the pageserver. This shifts decode CPU off the pageserver and reduces WAL fan-out per ingested byte.

TLA+ specification

safekeeper/spec/proto.tla and friends are TLA+ models of the consensus protocol. They encode the correctness properties (no committed record is ever lost; a committed record always survives any minority of failures) and were used to find a real bug during development. The Rust implementation is structured to mirror the spec's state transitions.

docs/safekeeper-protocol.md is the human-readable mapping between the TLA+ spec and the code.

Sharding and scale

A single safekeeper hosts many timelines from many tenants. A timeline can be moved between safekeepers via pull_timeline + a quorum reconfiguration (the storage controller can drive this). Sharding of a tenant across pageservers does not require sharding on safekeepers — the safekeeper protocol is per-timeline and is unaware of pageserver shards.

Hadron mode

safekeeper/src/hadron.rs is a specialized mode that combines a safekeeper and a small pageserver-like read path into one process for low-traffic tenants. It is opt-in and used for a subset of tenants in production.

Key source files

File	Purpose
`safekeeper/src/safekeeper.rs`	Consensus state machine.
`safekeeper/src/timeline.rs`	Per-timeline runtime.
`safekeeper/src/wal_storage.rs`	On-disk WAL ring.
`safekeeper/src/wal_backup.rs`	Remote-storage uploader.
`safekeeper/src/handler.rs`	libpq protocol handler.
`safekeeper/src/receive_wal.rs`	WAL receive side.
`safekeeper/src/send_wal.rs`	WAL send side.
`safekeeper/src/send_interpreted_wal.rs`	Interpreted WAL fan-out.
`safekeeper/src/broker.rs`	Storage-broker integration.
`safekeeper/src/control_file.rs`	Persistent header.
`safekeeper/spec/proto.tla`	TLA+ specification.
`docs/safekeeper-protocol.md`	Protocol writeup.
`docs/walservice.md`	Service overview.