WAL and recovery

Write-Ahead Logging is what makes PostgreSQL durable. Every change to a buffer page is logged to pg_wal/ before the change is flushed; on crash, recovery replays the log forward from the last checkpoint to rebuild the consistent state. WAL is also the substrate for streaming replication and logical decoding.

Source layout

src/backend/access/transam/
├── xlog.c          # the heart: state machine, write/flush, recovery driver
├── xloginsert.c    # XLogInsert API used by access methods
├── xlogarchive.c   # archive/restore handling
├── xlogfuncs.c     # SQL-callable functions (pg_switch_wal, pg_create_restore_point)
├── xlogprefetcher.c# read-ahead during recovery
├── xlogreader.c    # WAL parser
├── xlogrecovery.c  # the recovery state machine
├── xlogutils.c     # helpers
└── timeline.c      # WAL timelines (post-failover branching)

Plus per-rmgr WAL handlers in:

• src/backend/access/heap/heapam_xlog.c
• src/backend/access/nbtree/nbtxlog.c
• src/backend/access/gin/ginxlog.c
• ... one per access method, plus xact_redo for transaction commit/abort, dbase_redo for CREATE/DROP DATABASE, etc.

Key abstractions

Type	File	Role
XLogRecPtr (LSN)	src/include/access/xlogdefs.h	64-bit byte offset into the WAL stream. Pages carry the LSN of the last record that touched them.
XLogRecord	src/include/access/xlogrecord.h	Fixed-length WAL record header (LSN, length, xid, rmgr, info).
RmgrData	src/include/access/xlog_internal.h	Resource manager: redo, desc, identify, startup, cleanup, mask.
XLogReaderState	src/include/access/xlogreader.h	Iterator over WAL records, used by recovery and streaming.
XLogCtlData	src/backend/access/transam/xlog.c	Shared-memory state for WAL: insert pointer, write LSN, flush LSN.
RedoContext	xlogrecovery.c	State carried across redo calls.

Anatomy of a WAL record

Every record has:

• Header — type tag, length, owning XID, previous LSN.
• Block references — zero or more (rel, fork, blkno) tuples plus optional full-page images.
• Resource-manager data — opaque payload interpreted by the rmgr's redo function.

Producers fill these in via XLogBeginInsert, XLogRegisterBuffer, XLogRegisterData, then XLogInsert(rmgr, info). Source: src/backend/access/transam/xloginsert.c.

XLogBeginInsert();
XLogRegisterData((char *) &record_data, sizeof(record_data));
XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
recptr = XLogInsert(RM_HEAP_ID, XLOG_HEAP_INSERT);
PageSetLSN(BufferGetPage(buffer), recptr);

The resource managers are listed in src/include/access/rmgrlist.h:

PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL, xlog_decode)
PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, ...)
PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, ...)
PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, ...)
... etc.

A new access method that needs WAL must register its rmgr ID and provide redo (replay), desc (pretty-print), and a few hooks. Custom rmgrs from extensions are also possible (src/backend/access/transam/rmgr.c).

Write-ahead order

The WAL invariant is: a dirty page may not be flushed to disk before the WAL record describing the change has been flushed.

The buffer manager enforces this via XLogFlush(page->lsn) immediately before writing a buffer out (FlushBuffer in bufmgr.c). The WAL itself is written in 16 MB segment files under pg_wal/, named by LSN range.

Commit path

When a transaction commits:

sequenceDiagram
    participant Backend
    participant XLog as xlog.c
    participant Disk as pg_wal/
    participant CLOG

    Backend->>XLog: XLogInsert(XACT_COMMIT)
    XLog-->>Backend: commit LSN
    Backend->>XLog: XLogFlush(commit LSN)
    XLog->>Disk: write + fdatasync (synchronous_commit=on)
    Disk-->>XLog: ack
    Backend->>CLOG: TransactionIdCommitTree
    CLOG->>CLOG: mark XID committed (in shared SLRU)
    Backend->>Backend: ProcArrayEndTransaction

synchronous_commit controls when the flush happens:

• on — flush before reporting success (default).
• off — return immediately; the WAL writer flushes within wal_writer_delay. Crash may lose the last few transactions but never corrupts.
• local / remote_write / remote_apply — used with synchronous replication.

For low-latency commits, the commit group mechanism (in xact.c) lets multiple committing backends piggyback on one fsync.

Checkpoints

A checkpoint flushes every dirty buffer that was dirtied before a "redo" LSN, then writes a checkpoint record to WAL. Recovery only needs to replay from the redo LSN of the last checkpoint forward. Source: src/backend/access/transam/xlog.c::CreateCheckPoint, src/backend/postmaster/checkpointer.c.

Checkpoints are triggered by:

• Time (checkpoint_timeout, default 5 min).
• WAL volume (max_wal_size).
• Manual CHECKPOINT command.
• Shutdown / startup.

The checkpointer process spreads buffer writes over checkpoint_completion_target * checkpoint_timeout to smooth I/O.

Crash recovery

When the server starts and finds pg_control indicating an unclean shutdown, it enters recovery. Source: src/backend/access/transam/xlogrecovery.c.

graph TD
    Start --> ReadCtrl["read pg_control<br/>find latest checkpoint"]
    ReadCtrl --> OpenWAL["open WAL at checkpoint redo LSN"]
    OpenWAL --> ReadRec["xlog_reader: read next record"]
    ReadRec --> DispatchRedo["call rmgr->redo(record)"]
    DispatchRedo --> ApplyEffects["apply effects to buffers, CLOG, etc."]
    ApplyEffects --> Loop{more records<br/>and not yet at recovery target?}
    Loop -- "yes" --> ReadRec
    Loop -- "no" --> EndRecovery
    EndRecovery --> CreateCheckpoint["create new checkpoint"]
    CreateCheckpoint --> AcceptConnections

For each WAL record, the dispatcher calls RmgrTable[rmid].rm_redo(record), which mutates the relevant pages in the shared buffer pool. Pages with lsn >= record.lsn are skipped (already on disk in a newer state).

After replay, the startup process performs end-of-recovery cleanup: opens the WAL at the recovery point for new inserts, lets the background writer/checkpointer take over, and the postmaster opens the network listener.

Archive recovery and PITR

If archive_mode = on, the archiver process (src/backend/postmaster/pgarch.c) ships completed WAL segments to a long-term store (S3, NFS, etc.) via archive_command or archive_library. To restore:

1. Restore a base backup (taken with pg_basebackup).
2. Drop a recovery.signal file in the data directory.
3. Configure restore_command (and optionally recovery_target_*) in postgresql.conf.
4. Start the server.

The startup process pulls archived segments via restore_command and replays them. If recovery_target_time = '...' (or recovery_target_lsn, recovery_target_xid, recovery_target_name) is set, replay stops at that target. Otherwise it replays to the end of available WAL.

recovery.conf was deprecated in 12; recovery is now configured in postgresql.conf plus the signal file.

Timelines

Each time a cluster forks (e.g., promotion of a standby), it gets a new timeline ID — a monotonically increasing integer. WAL files are named by (timeline, segment), and recovery decides which timeline to follow at fork points. Source: src/backend/access/transam/timeline.c. The standby gets a new TLI when it gets promoted; pg_rewind aligns a former primary to the new timeline.

Hot standby

Streaming replicas can serve read-only queries while still applying WAL. To make this safe, the startup process:

• Updates the proc array with the WAL stream's running-xacts info.
• Enforces that user queries see only XIDs that are visible at the applied LSN.
• Pauses recovery briefly if a query holds a buffer pin that conflicts with a redo operation (or kills the query if max_standby_streaming_delay is exceeded).

Source: src/backend/storage/ipc/standby.c, src/backend/access/transam/xlogrecovery.c. The recovery conflict mechanism is one of the trickiest parts of the engine.

WAL prefetching

Recent versions added xlogprefetcher.c: while the startup process applies record N, a prefetcher reads ahead in the WAL stream and issues posix_fadvise/AIO reads for the pages records N+k will touch. Recovery throughput goes up substantially on cold-cache workloads. Controlled by recovery_prefetch.

Entry points for modification

• Adding WAL coverage to a new operation: call XLogBeginInsert + XLogRegisterBuffer + XLogInsert from your access method. Implement a redo function and add it to rmgrlist.h.
• Custom rmgrs from extensions: RegisterCustomRmgr in src/backend/access/transam/rmgr.c.
• Tweaking checkpoint behavior: src/backend/postmaster/checkpointer.c plus the shared-state in xlog.c.
• Recovery hooks: standby info messages and recovery target handling live in xlogrecovery.c.

For the consumers of WAL — replication and CDC — see Replication. For the transaction state above WAL, see MVCC and transactions.