Chunk encodings

tsdb/chunkenc/ implements the on-the-wire chunk formats used by the head and the blocks. Each chunk encodes a sequence of (t, v) samples for a single series; queries iterate over chunks and decode them into the streaming sample iterators that the PromQL engine consumes.

Encoding catalogue

Encoding	File	Used for
EncXOR (Gorilla XOR)	xor.go, bstream.go	Float samples (the original Gorilla compression scheme).
EncXOR2 (3.x)	xor2.go	Float samples with explicit start timestamp; replacement for EncXOR for scraped data.
EncHistogram	histogram.go, histogram_meta.go	Native integer histograms.
EncFloatHistogram	float_histogram.go	Native float histograms.
Varbit	varbit.go	Variable-bit-width primitive used by histogram bucket counts.

Encoding constants and the Chunk/Iterator/Appender interfaces are in tsdb/chunkenc/chunk.go.

Gorilla XOR (EncXOR)

The original Prometheus chunk format, taken from Facebook's Gorilla paper. Compresses a stream of float64 samples by:

• Delta-of-delta encoding for timestamps.
• XOR encoding for values: identical samples take 1 bit, value changes encode the leading/trailing zero count plus the meaningful bits.

Implementation lives in tsdb/chunkenc/xor.go. The bstream (bstream.go) is the bit-level reader/writer used by both XOR and varbit.

XOR2 (EncXOR2)

xor2.go introduces a second-generation XOR chunk that stores an explicit start timestamp at chunk start (and only encodes deltas afterwards). This is a prerequisite for storing start timestamps alongside samples without a separate record type.

Currently behind --enable-feature=xor2-encoding (3.11, #18062). Once enabled, the format is fully readable on read paths in 3.10+ TSDBs (downgrade safe). The plan is to make XOR2 the default once it has been deployed widely.

Native histograms

histogram.go (and float_histogram.go) encode a sparse, exponential-bucket histogram per sample. Each chunk stores a sequence of histograms with delta encoding between consecutive layouts.

Key concepts:

• Schema — a positive integer that sets the bucket boundary spacing (schema=2 -> ratio 2^(2^-2) = 1.189...).
• Zero bucket — special bucket counting samples whose absolute value is below ZeroThreshold.
• Positive/negative buckets — sparse arrays indexed by exponent.
• Bucket diffs — delta-encoded across consecutive samples within a chunk.

Cross-chunk schema changes force a new chunk; intra-chunk schema is constant.

histogram_meta.go validates layout changes (e.g. addition of a new bucket span) and is the source of the validation rules used during ingestion.

Custom-bucket histograms (NHCB)

NHCB uses the native histogram chunk format but with explicit, non-exponential bucket boundaries. The translation from classical (cumulative) histograms to NHCB lives in util/convertnhcb/.

Iterating chunks

chunk.Appender writes samples; chunk.Iterator walks them:

it := chunk.Iterator(nil) // pass an existing iterator to reuse buffers
for vt := it.Next(); vt != chunkenc.ValNone; vt = it.Next() {
    switch vt {
    case chunkenc.ValFloat:
        t, v := it.At()
    case chunkenc.ValHistogram:
        t, h := it.AtHistogram(nil)
    case chunkenc.ValFloatHistogram:
        t, fh := it.AtFloatHistogram(nil)
    }
}

The iterator interface mirrors storage.SeriesIterator so chunks can plug straight into the engine.

Cross-chunk merging

chunkenc.Pool (in chunk.go) backs a sync.Pool per encoding, used heavily by readers to reuse iterators. The MergeIterator (tsdb/chunkenc/...) deduplicates and merges samples across overlapping chunks (e.g. during vertical compaction or out-of-order merging at read time).

Chunk size and rotation

In the head:

• A chunk holds at most MaxBytesPerXORChunk bytes (default 1 KB ≈ 240 samples) or SamplesPerChunk (default 120) — whichever is smaller.
• Histograms have analogous limits.
• When rotated, the chunk is mmapped to chunks_head/NNNNNNN.

In blocks, chunks are densely packed into segment files of up to 512 MB.

Floating-point precision

float_histogram.go allows fractional bucket counts (necessary after rate() and binary operators on histograms). quantile.go (in promql/) consumes float histograms when computing histogram_quantile().

Entry points for modification

• New chunk encoding: add an Encoding constant in chunk.go, implement Chunk/Appender/Iterator, register a Pool, and update the chunk reader's switch statement. Document the format inline.
• Tweak XOR layout: xor.go and xor2.go should remain wire-compatible with previous Prometheus releases.
• Histogram changes: histogram_meta.go is the validation hot path; benchmark before merging.

See Head for chunk lifecycle within the head and Blocks for how chunks are persisted to segment files.