WAL and checkpointing

The Write-Ahead Log (WAL) under tsdb/wlog/ is the durable, append-only log that backs both the server head and the agent. It is the single source of truth for recovery and for remote write tailing.

Layout on disk

data/
└── wal/
    ├── 00000000              # closed segment, ~128 MB
    ├── 00000001              # …
    ├── checkpoint.00000003/  # produced when the watcher catches up + truncate
    │   └── 00000000
    └── 00000004              # current open segment

Defaults:

• Segment size: 128 MB (--storage.tsdb.wal-segment-size).
• Compression: none by default; snappy and zstd available via --storage.tsdb.wal-compression-type.
• Sync: --storage.tsdb.wal-sync-strategy (always|never|periodic); periodic uses a 1s flush interval.

Key types

Type	File	Role
WL	tsdb/wlog/wlog.go	Top-level WAL struct: segment writer, mutex, metrics.
Reader	tsdb/wlog/reader.go	Block-by-block reader for replay.
LiveReader	tsdb/wlog/live_reader.go	Tail reader that follows growing segments (used by remote write).
Watcher	tsdb/wlog/watcher.go	Subscriber abstraction for tailing the WAL.
Checkpoint	tsdb/wlog/checkpoint.go	Compacted prefix produced when truncating.
RecordEnc/RecordDec	tsdb/record/record.go	Encode/decode the typed records (Series, Samples, Tombstones, Exemplars, …).

Record types

Defined in tsdb/record/record.go:

Type	What it carries
RecordSeries	New series + label set + ref.
RecordSamples	Batched float samples.
RecordTombstones	Deletion ranges.
RecordExemplars	Exemplars attached to existing series.
RecordHistogramSamples / RecordFloatHistogramSamples	Native (float) histogram samples.
RecordMetadata	Type/unit/help updates for a series.
RecordCustomBucketsHistogramSamples	NHCB samples.
RecordST (3.11)	Start-timestamp records (when --enable-feature=st-storage).

The record format begins with a single-byte type tag followed by varint-encoded payload. The decoder is hot — it must avoid allocations.

Replay on startup

Head.Init() opens the WAL and walks every segment:

graph LR
    A[Open WAL] --> B[Read checkpoint dir]
    B --> C[Replay records]
    C --> D[Re-create memSeries]
    C --> E[Re-add samples to in-memory chunks]
    E --> F[Mmap leftover head chunks]
    F --> G[Set replay-status to done]

Concurrent replay uses up to runtime.GOMAXPROCS(0) workers (defaultWALReplayConcurrency in tsdb/head.go). Progress is exposed at /api/v1/status/walreplay. Corrupt segments are repaired by tsdb/repair.go (truncate to the last valid record), bumping prometheus_tsdb_corruptions_total.

Truncation and checkpoints

After a block is compacted, Head.Truncate(t) removes all in-memory state older than t and tells the WAL it can drop everything older. The WAL produces a checkpoint: a new directory checkpoint.NNNNNNN/ containing only the records still relevant (active series, undelivered samples). After the checkpoint is fsync'd, the older segments are deleted.

Checkpoint logic is in tsdb/wlog/checkpoint.go. The KeepFrom parameter and the watcher set determine which records survive; remote write watchers can hold the checkpoint behind their last-replicated position.

prometheus_tsdb_checkpoint_* metrics expose duration, deleted records, kept records, and failures.

Watching for remote write

storage/remote/wal_watcher (in storage/remote/queue_manager.go) uses wlog.Watcher with a LiveReader to tail the WAL in real time and forward samples to the queue manager. The watcher:

• Tracks per-shard last-replicated offsets in <wal>/remote/<remote_name>/.
• Replays unsent records on startup (the watcher walks back to the offset, then catches up).
• Honours WAL truncation barriers — checkpoints are not deleted while a watcher still trails the segment they cover.

Optimisations in #18250 (3.11) reuse internal buffers and avoid per-record allocations on the watcher hot path.

Agent-mode WAL

tsdb/agent/db.go reuses the same wlog package but never compacts to blocks. Its truncation policy is age-based (--storage.agent.retention.min-time / .max-time). Replay on startup is similar but skips chunk allocation — the agent has no head chunks beyond what the WAL contains.

Failure modes and metrics

• prometheus_tsdb_wal_corruptions_total — segment had an invalid record at replay; the offset was truncated.
• prometheus_tsdb_wal_truncate_duration_seconds — duration of Truncate calls.
• prometheus_tsdb_wal_writes_failed_total — write errors (disk full, permission denied).
• prometheus_tsdb_wal_completed_pages_total — page-level write counter; useful when chasing fsync regressions.

If the WAL directory is on a slow disk, prometheus_tsdb_data_replay_duration_seconds will be the dominant component of startup time. The 3.11 --enable-feature=fast-startup writes a series_state.json file alongside the WAL to pre-load active series state.

Entry points for modification

• New record type: add a constant to tsdb/record/record.go, encoding/decoding helpers, and a handler in head_wal.go. Update the watcher in tsdb/wlog/watcher.go to forward the new type.
• Compression strategy: add a new compression type in util/compression/ and wire it through wlog.WL.
• Truncation policy: the Head.Truncate and checkpoint logic interact carefully with the watcher set — touch with care.

See TSDB index and Blocks for the rest of the storage stack.