Remote write, remote read, and OTLP

Active contributors: cstyan, bwplotka, tomwilkie, alexgreenbank, ArthurSens, jesusvazquez

Purpose

storage/remote/ is the largest sibling of tsdb/. It implements:

• Outbound remote write — ship samples to a remote storage backend over HTTP.
• Inbound remote write receiver — accept v1 or v2 protobuf payloads from another Prometheus or compatible sender.
• OTLP receiver — accept OpenTelemetry Protocol metrics and translate them into Prometheus's internal model.
• Remote read — pull historical series from a remote backend as if they were local.
• AzureAD / Google IAM helpers — pluggable HTTP client decorators for cloud-managed identity.

Directory layout

storage/remote/
├── storage.go            # remote.Storage: top-level Storage that wraps queues + read clients.
├── queue_manager.go      # The QueueManager: shards, batching, send loop, backoff (~2,300 lines).
├── stats.go              # EWMA + sharding heuristics.
├── ewma.go               # Exponentially weighted moving average implementation.
├── intern.go             # String interning for label values.
├── max_timestamp.go      # Atomic max-timestamp tracker.
├── client.go             # HTTP client used by writes.
├── codec.go              # Protobuf <-> internal format conversions.
├── chunked.go            # Chunked HTTP response framing for streaming reads.
├── dial_context.go       # Custom dial helpers (TLS, proxy).
├── metadata_watcher.go   # Forwards metric metadata to remote write v1.
├── read.go / read_handler.go      # Remote read client + server handlers.
├── write.go / write_handler.go    # Outbound + inbound write.
├── write_otlp_handler.go          # OTLP receiver.
├── azuread/                       # Azure managed identity auth.
├── googleiam/                     # GCE / GKE managed identity auth.
└── otlptranslator/                # OTLP -> Prometheus translation (delegated to prometheusremotewrite).

Outbound remote write

sequenceDiagram
    participant Append as Storage Appender
    participant QM as QueueManager
    participant Watcher as wlog.Watcher
    participant Shards as N shards
    participant Receiver

    Note over Append,Watcher: Two ingest paths feed the queue
    Append-->>QM: Append(sample)
    Watcher->>QM: walRecord (server mode)
    QM->>Shards: hash(seriesRef) % N
    Shards->>Shards: build batch (size or time)
    Shards->>Receiver: HTTP POST /api/v1/write (v1)<br/>or /api/v1/write?version=2 (v2)
    Receiver-->>Shards: 2xx / 4xx / 5xx
    QM->>QM: shard count adjustment (EWMA)

QueueManager (in queue_manager.go) is the heart. Each RemoteWriteConfig produces one queue. Key behaviours:

• Sharding: series are hashed into N shards, sized by min_shards <= N <= max_shards. The shard count is adjusted every 10 seconds based on an EWMA of the sample backlog (shardUpdateDuration, shardToleranceFraction = 0.3).
• Batching: each shard buffers samples until either max_samples_per_send (default 2000) or batch_send_deadline (default 5s).
• Compression: snappy is used for v1; v2 supports the same compression set as the WAL (util/compression).
• Backpressure: if all shards are full, QueueManager.Append returns false and drops the sample, incrementing prometheus_remote_storage_samples_dropped_total{reason="too_old"|"queue_full"|...}.
• Retry: transient (5xx, network) errors retry with exponential backoff. 4xx other than 429 fail-fast, dropping the batch.

In server mode, the WAL watcher (tsdb/wlog/watcher.go) reads the WAL forwards and feeds the queue manager so a sample sent via remote write is durable on disk before it is enqueued. In agent mode, the same watcher is the only path.

Remote write protocol versions

Version	Wire format	Notes
v1.0	prompb.WriteRequest (snappy)	Original; samples + optional metadata via MetadataWatcher.
v2 (RW2)	prompb/io/prometheus/write/v2.Request	Per-sample exemplars, native histograms, NHCB, types/units in-band.

The receiver discovers what message types the sender supports via the X-Prometheus-Remote-Write-Version header and a content-type negotiation.

OTLP receiver

/api/v1/otlp/v1/metrics (POST) accepts OpenTelemetry metric exports. The handler in write_otlp_handler.go translates each OTLP sum/gauge/histogram/summary into a Prometheus appender call via the otlptranslator/ sub-package, which delegates to the upstream github.com/prometheus/otlptranslator library.

OTLP-specific behaviour:

• Delta -> cumulative conversion is opt-in (--storage.otlp.convert-delta flag, behind --enable-feature=otlp-deltatocumulative).
• Native delta ingestion (preserve delta temporality) is opt-in (--enable-feature=otlp-native-delta-ingestion).
• Lookback delta is honoured during conversion.
• The __name__ attribute is filtered out to prevent duplicate labels (3.10 fix #17917).

Remote read

The remote read protocol is the inverse: a Prometheus issues ReadRequests with a label matcher and time range, and the remote endpoint streams back chunks. Two transports:

• Buffered protobuf (legacy, read_handler.go): one big response.
• Chunked streaming (chunked.go): application/x-streamed-protobuf;proto=prometheus.ChunkedReadResponse — interleaved chunks.

Local Prometheus exposes the read endpoint at /api/v1/read (used by other Prometheus instances or Mimir/Thanos for federation). The client lives in read.go.

OTLP translator

storage/remote/otlptranslator/prometheusremotewrite/ is a thin wrapper around the upstream github.com/prometheus/otlptranslator library. The translator owns:

• Metric name sanitisation (UTF-8 to Prometheus-friendly).
• Label name escaping.
• Histogram bucket conversion (classical to native).
• Summary expansion (sum/count/quantile).

Self-metrics

Per-queue (label remote_name, url):

• prometheus_remote_storage_samples_in_total
• prometheus_remote_storage_samples_pending
• prometheus_remote_storage_samples_failed_total{reason}
• prometheus_remote_storage_samples_retried_total
• prometheus_remote_storage_samples_dropped_total{reason}
• prometheus_remote_storage_shards{state} — desired, min, max
• prometheus_remote_storage_shard_capacity
• prometheus_remote_storage_sent_batch_duration_seconds
• prometheus_remote_storage_highest_timestamp_in_seconds
• prometheus_remote_storage_string_interner_zero_reference_releases_total

Inbound (receiver):

• prometheus_remote_storage_received_samples_total
• prometheus_remote_storage_received_histograms_total

Configuration surface

remote_write:
  - url: https://mimir.example.com/api/v1/push
    name: mimir
    queue_config:
      capacity: 2500
      max_samples_per_send: 2000
      max_shards: 200
      min_shards: 1
      batch_send_deadline: 5s
    metadata_config:
      send: true
      send_interval: 1m
    write_relabel_configs:
      - action: drop
        source_labels: [__name__]
        regex: 'noisy_metric.*'
    sigv4: { region: us-east-1 } # AWS sigv4 auth

CLI flags surface OTLP-specific behaviours:

• --web.enable-otlp-receiver — enable the OTLP endpoint.
• --storage.otlp.translation-strategy=NoUTF8EscapingWithSuffixes|UnderscoreEscapingWithSuffixes (3.x).

Entry points for modification

• Tweak shard heuristics: queue_manager.go::calculateDesiredShards. Performance changes need before/after prombench runs.
• Add an auth method: create a sub-package similar to azuread/ and register it in storage.go::ApplyConfig.
• OTLP translation rule: the translator is an external library; for in-tree behaviour (delta/cumulative, naming), edit write_otlp_handler.go and the option struct.

See Storage for the interface contracts and Remote write feature page for the operational view.