Storage

Purpose

storage/ defines the interfaces every storage backend must implement and provides a fanout that lets the server multiplex writes to the local TSDB and any number of remote write queues. The interfaces live here; the implementations live in tsdb/, tsdb/agent/, and storage/remote/.

Files

storage/
├── interface.go          # Storage, Queryable, Appendable, ChunkQueryable, ExemplarStorage.
├── interface_append.go   # AppenderV2 (in-progress migration).
├── fanout.go             # fanoutStorage: writes to primary + secondaries, queries the union.
├── secondary.go          # Wraps a storage that may fail without breaking the query.
├── series.go             # Generic series helpers (chain, sort, merge).
├── merge.go              # MergeSeriesSet for read-time merging.
├── memoized_iterator.go  # Buffered iterator wrapper.
├── lazy.go               # Lazy series.
├── noop.go               # Empty storage (used in tests, agent mode UI).
├── generic.go            # Type-parametrised iterator helpers.
├── buffer.go             # Per-series buffered samples.
├── errors.go             # Common error types.
└── remote/               # Remote write + read + OTLP.

The Storage interface

type Storage interface {
    SampleAndChunkQueryable
    Appendable      // V1 appender (being phased out)
    AppendableV2    // V2 appender (in progress)
    StartTime() (int64, error)
    Close() error
}

Implementations:

• tsdb.DB — full local TSDB.
• tsdb/agent.DB — WAL-only, agent mode.
• fanoutStorage (storage/fanout.go) — composite that writes to a primary + secondaries.
• noopStorage (storage/noop.go) — for tests and agent-mode UI degraded paths.

V1 vs V2 appenders

The migration tracker is #17632; ETA: removal of the V1 appender by Q2 2026. Differences summarised in Glossary. Subsystems that have been migrated to write through V2:

• scrape/scrape_append_v2.go
• storage/remote/write_handler.go (RW2 receiver)
• storage/remote/write_otlp_handler.go (OTLP receiver)
• tsdb/head_append_v2.go (TSDB)
• tsdb/agent/db_append_v2.go (agent)

The storage/teststorage helper supports both shapes for tests.

Fanout storage

storage/fanout.go::fanoutStorage writes appenders into the primary first; on success, it writes to each secondary. Failures on secondaries are logged but do not fail the primary append (so the local TSDB stays authoritative). On read, fanoutQuerier queries primaries + secondaries and merges results via merge.go::NewMergeSeriesSet.

In cmd/prometheus/main.go, the fanout is constructed as fanoutStorage(localStorage, remoteStorage) where remoteStorage is the storage/remote.Storage aggregator.

Errors

storage/interface.go defines the canonical error sentinels:

• ErrOutOfOrderSample, ErrOutOfBounds, ErrTooOldSample
• ErrDuplicateSampleForTimestamp
• ErrOutOfOrderExemplar, ErrDuplicateExemplar, ErrExemplarLabelLength, ErrExemplarsDisabled
• ErrNativeHistogramsDisabled
• ErrOutOfOrderST, ErrSTNewerThanSample

Receivers and parsers should errors.Is against these (don't string-match).

Exemplar storage

Exemplars are append-only structures attached to specific series. storage.ExemplarStorage (in interface.go) defines AppendExemplar, ExemplarQueryable. Implementations:

• tsdb/exemplar.go — bounded circular buffer per series, sized by --storage.exemplars.max-exemplars.

Series sets and iterators

Common helpers used by both the TSDB and the remote read code:

• MergeSeriesSet — N-way merge of pre-sorted sets.
• NewSeriesSetFromChunkSeriesSet — adapt chunk-level to sample-level sets.
• BufferedSeriesIterator — bounded look-back during PromQL evaluation.
• MemoizedSeriesIterator — single-sample memoisation across calls.

Annotations

storage.SeriesSet and storage.Querier return optional annotations.Annotations warnings. Examples: PromQLInfo: histogram bucket monotonicity violation, PromQLWarning: gauge with no recent samples. Annotations propagate up to PromQL Result.Warnings / Result.Infos and surface in the UI.

Integration points

• TSDB / Agent: primary implementations.
• Remote write/read: storage/remote.Storage is constructed in cmd/prometheus/main.go and registered as a fanout secondary.
• PromQL: the engine consumes storage.Queryable + storage.ChunkQueryable.
• Web API: the same interfaces drive /api/v1/series, /api/v1/labels, etc.

Entry points for modification

• New storage backend: implement Storage (or a subset; the agent gets away with Appendable + StartTime + Close).
• New appender capability: prefer AppenderV2 since V1 is being removed.
• Fanout policy change: storage/fanout.go is sensitive to write-failure semantics. Reviewers will ask for a clear durability story.

See TSDB and Remote write/read for the major implementations.