TSDB

Active contributors: jesusvazquez, codesome, bwplotka, krajorama, kgeckhart

Purpose

The Prometheus time series database — tsdb/ — is a self-contained, append-mostly database tailored to monitoring data. It writes a Write-Ahead Log to disk for durability, keeps the most recent ~2 hours of samples in an in-memory Head, periodically compacts the head into immutable on-disk blocks, and provides ranged queries over the union of head and block data.

It is also reusable: it ships its own tsdb/CHANGELOG.md, has independent users, and is consumed via the Storage interface (storage/interface.go).

Directory layout

tsdb/
├── db.go                  # tsdb.DB: open/close, blocks, compaction, retention.
├── head.go                # In-memory Head: stripe locks, series tracker, exemplars.
├── head_append.go         # Append path V1 (Appender contract).
├── head_append_v2.go      # Append path V2 (AppendEntry contract).
├── head_read.go           # Read path against the Head.
├── head_wal.go            # WAL replay during DB open.
├── ooo_head.go            # Out-of-order head + queries.
├── block.go               # On-disk block lifecycle (open, write, delete).
├── compact.go             # Compactor: leveled compaction, vertical (overlap) compaction.
├── querier.go             # block + head queriers, fanout.
├── exemplar.go            # In-memory exemplar storage.
├── isolation.go           # Lockless read isolation.
├── repair.go              # WAL repair on startup.
├── chunkenc/              # Chunk encodings: XOR, XOR2, histograms, varbit.
├── chunks/                # Persistent chunk files (chunks_head + segment files).
├── index/                 # Block index: postings, label index.
├── wlog/                  # Write-ahead log primitives.
├── record/                # WAL record types and codecs.
├── tombstones/            # Block-level deletion records.
├── encoding/              # Generic varint / byte encoding helpers.
├── compression/           # Optional WAL compression types.
├── fileutil/              # Mmap, file locking, fadvise.
├── agent/                 # Agent-mode WAL-only storage.
├── docs/                  # In-tree design notes (head_chunks, ooo, agent).
└── testdata/              # Fixture blocks and large series files.

How a sample becomes durable

sequenceDiagram
    participant Caller
    participant Head as Head
    participant WAL as wlog
    participant CHW as chunks_head
    participant Block as Block(s)

    Caller->>Head: Appender.Append(seriesRef, ts, value)
    Note right of Head: Buffered in headAppender.<br/>No lock held.
    Caller->>Head: Appender.Commit()
    Head->>WAL: Log() series records + samples
    Head->>Head: Insert into stripe-locked memSeries
    Note right of Head: After ~2h of head time:
    Head->>CHW: Cut chunk, mmap to chunks_head/NNNNNNN
    Note right of Head: At GC interval:
    Head->>Block: Compact head -> ./<ULID>/
    Head->>WAL: Truncate + checkpoint

The append path's hot logic lives in tsdb/head_append.go (V1) and tsdb/head_append_v2.go (V2). V2 supports start timestamps, per-sample exemplars, always-on metadata, and a single AppendEntry value — see Storage for the migration tracker.

Sub-pages

The TSDB is large enough to warrant focused pages:

• Head and append path — Head, memSeries, V1/V2 append, isolation.
• WAL and checkpointing — tsdb/wlog/, segment lifecycle, replay.
• Blocks and compaction — Block, Compactor, retention, vertical compaction.
• Index and postings — tsdb/index/, posting lists, label index.
• Chunk encodings — tsdb/chunkenc/, XOR, XOR2, histograms, varbit.

Configuration surface

Most knobs are CLI flags forwarded into tsdb.Options (see tsdb/db.go::DefaultOptions):

• --storage.tsdb.path — data directory.
• --storage.tsdb.retention.time / .size / .percentage — retention by age, byte size (3.11), or disk percentage.
• --storage.tsdb.no-lockfile — disable the dir lockfile.
• --storage.tsdb.head-chunks-write-buffer-size / .head-chunks-write-queue-size.
• --storage.tsdb.wal-compression-type — none|snappy|zstd.
• --storage.tsdb.wal-segment-size.
• --storage.tsdb.allow-overlapping-compaction — disable vertical compaction.
• --storage.tsdb.head-mmap-budget / .head-mmap-batch-size (3.x).
• --enable-feature=memory-snapshot-on-shutdown / out-of-order-attributes / xor2-encoding / st-storage / fast-startup.

The complete list with defaults is at cmd/prometheus/main.go and the per-config-file equivalents are at config/config.go::TSDBConfig.

Self-metrics

All metrics are prefixed prometheus_tsdb_*. Notable counters and gauges:

• prometheus_tsdb_head_series — number of active series in the head.
• prometheus_tsdb_head_samples_appended_total — append throughput.
• prometheus_tsdb_compaction_* — compaction duration, attempts, populating block.
• prometheus_tsdb_wal_* — WAL truncation, checkpoint, corruption.
• prometheus_tsdb_storage_blocks_bytes — disk usage.
• prometheus_tsdb_data_replay_duration_seconds — WAL replay duration on startup.
• prometheus_tsdb_isolation_* — isolation high-water marks.

Integration points

• The TSDB is the default backing store for storage.Storage (storage/fanout.go wraps it).
• Remote write tails the WAL via tsdb/wlog/watcher.go to ship samples without going through the storage interface again.
• The exemplar store (tsdb/exemplar.go) is reachable via storage.ExemplarQueryable.
• cmd/promtool/tsdb.go opens a TSDB read-only for offline analysis.

Entry points for modification

• New on-disk format: every change must be readable by tsdb.DB.Open for at least one minor version. tsdb/CHANGELOG.md tracks compatibility windows.
• New chunk encoding: add a type to tsdb/chunkenc/ and a constant in tsdb/chunkenc/chunk.go. The XOR2 chunk (xor2.go) is the model for new formats with start timestamps.
• New WAL record: add a record type to tsdb/record/record.go and corresponding decoder in wlog/watcher.go (so remote write can replay it).
• Compaction tuning: the planner is in tsdb/compact.go; changes here have outsize effects on disk space and read latency, so include benchmark numbers and a prombench run in your PR.