Exemplars

Exemplars are pointers from a sample to a piece of context outside the metric set — typically a trace ID. They are stored in a separate, bounded circular buffer per series and exposed via PromQL and the API.

Anatomy

type Exemplar struct {
    Labels labels.Labels  // typically {trace_id="..."}
    Value  float64
    Ts     int64
    HasTs  bool
}

labels.Labels total length is capped at ExemplarMaxLabelSetLength (128 bytes). The TSDB rejects oversize exemplars with ErrExemplarLabelLength.

Lifecycle

graph LR
    Scrape[scrape parser] -->|attached to sample| App[Appender]
    App -->|AppendExemplar| ES[exemplar store]
    ES -->|circular buffer| MM[in-memory ring]
    Q[/api/v1/query_exemplars] -->|read| ES

The store (tsdb/exemplar.go) keeps a fixed-size buffer sized by --storage.exemplars.max-exemplars (or the YAML equivalent under storage.exemplars). When full, the oldest exemplars are evicted.

Exemplars are also written to the WAL (tsdb/record/record.go::ExemplarRecord) so they survive a restart for the WAL retention window.

Enabling

CLI flag: --enable-feature=exemplar-storage. Without it, exemplar storage is disabled and ErrExemplarsDisabled is returned. With it:

• Scrape parsers extract exemplars from OpenMetrics and protobuf payloads.
• Remote write v2 carries exemplars per sample.
• The query endpoint returns them.

API

GET /api/v1/query_exemplars?query=<expr>&start=<t>&end=<t> returns matching exemplars. Response shape:

{
  "status": "success",
  "data": [
    {
      "seriesLabels": { "__name__": "http_request_duration_seconds_bucket", ... },
      "exemplars": [
        { "labels": { "trace_id": "abc123" }, "value": 0.123, "timestamp": 1700000000.123 }
      ]
    }
  ]
}

UI integration

The Mantine UI's graph component fetches exemplars alongside series and overlays them as small markers. Hovering an exemplar shows the trace ID; clicking it navigates to a configurable URL template (web.external_url + custom format).

Recent fixes

• 3.10 #17863 — fix exemplars being incorrectly discarded during exemplar buffer grow/shrink.
• 3.11 #18056 — OTLP exemplars getting mixed between incorrect parts of a histogram (fixed in the OTLP receiver).

Entry points for modification

• Bigger or smaller buffer: --storage.exemplars.max-exemplars. Per-series budget is implicit.
• New transport: RW1 doesn't carry exemplars; RW2 does. The OTLP receiver emits them via the V2 appender.
• Eviction policy: tsdb/exemplar.go::circularBufferAppendable.AppendExemplar is the place to alter LRU vs FIFO.

See Storage and TSDB.