Remote write

Remote write is the outbound protocol that ships samples to a long-term storage backend over HTTP. This page is the operational view; the implementation is at Remote write/read.

When to use it

• Long-term storage (data retention beyond what local TSDB can keep on disk).
• Cluster-wide aggregation across many Prometheus instances.
• Cross-region replication.
• HA pairs that share data through a remote backend.

Configuration

remote_write:
  - url: https://mimir.example.com/api/v1/push
    name: mimir-prod
    queue_config:
      capacity: 5000
      max_samples_per_send: 2000
      max_shards: 200
      min_shards: 1
      batch_send_deadline: 5s
      retry_on_http_429: true
      min_backoff: 30ms
      max_backoff: 5s
    metadata_config:
      send: true
      send_interval: 1m
    write_relabel_configs:
      - action: drop
        source_labels: [__name__]
        regex: 'go_gc_.*'
    sigv4: { region: eu-west-1 }
    headers:
      X-Scope-OrgID: 'tenant-1'
    proto_message: io.prometheus.write.v2.Request # opt into RW2
    send_native_histograms: true
    send_exemplars: true

The full schema is in config/config.go::RemoteWriteConfig.

Protocol versions

• RW1 — prompb.WriteRequest, snappy. Default.
• RW2 — prompb/io/prometheus/write/v2.Request. Bundles type/unit metadata, exemplars per sample, and native histograms in a single payload. Receivers indicate support via the Accept-Remote-Write-Versions header.

The sender configures proto_message: to opt into RW2. The receiver must declare acceptRemoteWriteProtoMsgs in its API constructor.

Sender architecture

The sender lives in storage/remote/queue_manager.go:

• Storage aggregates one QueueManager per RemoteWriteConfig.
• The WAL watcher (tsdb/wlog/watcher.go) tails the local WAL and feeds the queue manager.
• The queue manager hashes series across N shards and dispatches each batch to a goroutine.
• Shard count auto-scales every 10s using an EWMA over backlog.

Samples can be dropped when the queue is full; counters: prometheus_remote_storage_samples_dropped_total{reason}.

Sender troubleshooting

Symptom	Where to look
Backlog growing	Increase max_shards; check receiver latency.
Bursty 429s	retry_on_http_429: true; tune max_backoff.
Many 4xx errors	Check request body: receiver may reject due to OOO, label limits.
Drops with reason="too_old"	Receiver lookback window too small; or local clock skew.
samples_pending flatlined high	Receiver healthy but slow; check sent_batch_duration_seconds.
samples_dropped_total{reason="dropped_series"}	write_relabel_configs actively dropping things.

The 3.11 #18214 fix corrects prometheus_remote_storage_sent_batch_duration_seconds so it now measures the actual HTTP send rather than the time before send.

Receiver

/api/v1/write accepts inbound RW1 / RW2 payloads when --web.enable-remote-write-receiver is set. The handler is storage/remote/write_handler.go. Out-of-order samples are accepted within out_of_order_time_window; older are rejected.

The handler validates label names, applies built-in limits (max_samples_per_send, max_query_length, etc.), and routes through the AppenderV2 path.

The 3.11 fix #18084 corrected ErrTooOldSample returning HTTP 500 instead of 400 in PRW v2 histogram write paths — a sender retrying on 5xx would otherwise loop indefinitely.

Authentication

Remote write supports:

• HTTP basic auth.
• Bearer token (file or inline).
• TLS client certificates.
• AWS SigV4 (sigv4:).
• Azure managed identity (azuread:).
• Google IAM (googleiam:).
• OAuth2.

These are decorations of the same *http.Client; their implementations are in storage/remote/azuread/, googleiam/, and prometheus/sigv4.

Entry points for modification

• New auth scheme: create a sub-package under storage/remote/<scheme>/ and register it in storage.go::ApplyConfig.
• Tweak shard auto-scaling: queue_manager.go::calculateDesiredShards.
• New receiver behaviour: edit write_handler.go. Fields are part of the public protocol, so be backward-compatible.

See Remote read for the inverse direction and OTLP for the OTLP receiver.