Remote write and remote read

The HTTP+protobuf protocols Prometheus uses to ship samples to and pull samples from a remote backend.

Remote write

POST /api/v1/write
Content-Type: application/x-protobuf
Content-Encoding: snappy
X-Prometheus-Remote-Write-Version: 0.1.0

For RW2:

Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request
X-Prometheus-Remote-Write-Version: 2.0.0

The body is a prompb.WriteRequest (RW1) or io.prometheus.write.v2.Request (RW2) compressed with snappy. Both schemas are defined in prompb/.

The receiver is storage/remote/write_handler.go. It validates label names, applies the configured ingestion limits, and routes samples through the AppenderV2 path. Any errors are returned as HTTP status codes:

• 200/204 — success.
• 400 — bad request (invalid labels, oversize labels, malformed body, OOO when not allowed).
• 408 — request timeout.
• 409 — duplicate sample for timestamp.
• 413 — body size exceeds limit.
• 429 — too many requests; senders retry with backoff.
• 500 — unexpected server error.

The 3.11 fix #18084 corrected the OOO histogram path returning 500 instead of 400, which previously caused infinite client retries.

Remote read

POST /api/v1/read
Content-Type: application/x-protobuf
Accept: application/x-streamed-protobuf;proto=prometheus.ChunkedReadResponse, application/x-protobuf
X-Prometheus-Remote-Read-Version: 0.1.0

The body is a prompb.ReadRequest containing one or more queries (matchers + time range). The response is either:

• A single buffered prompb.ReadResponse (legacy), or
• A stream of prompb.ChunkedReadResponse frames (preferred), each ~1 MB.

The handler in storage/remote/read_handler.go enforces:

• --storage.remote.read-sample-limit — total samples returned across the response.
• --storage.remote.read-concurrent-limit — concurrent in-flight read requests.
• --storage.remote.read-max-bytes-in-frame — chunked frame size budget.

Authentication

Both endpoints sit behind the standard web layer. TLS, basic auth, and bearer tokens are configured via --web.config.file (the prometheus/exporter-toolkit/web config schema).

For senders, remote write configs in prometheus.yml support: basic_auth, bearer_token / bearer_token_file, oauth2, tls_config, sigv4, azuread, googleiam, plus arbitrary headers.

Recent protocol-level fixes

• 3.11 #18084 — return 400 (not 500) on OOO in PRW v2 histogram paths.
• 3.11 #18214 — fix prometheus_remote_storage_sent_batch_duration_seconds measuring before the request was sent.
• 3.11 #18250 — performance improvement: reuse internal buffers in WAL watching.
• 3.10 #17849 — Remote Read response framing improvements.
• 3.10 — Allow snappy compression in writes for OTLP.

Compliance test suite

The compliance/ Go module is a separate test runner that validates the Remote Write 1.0 spec against any sender implementation. Run with make -C compliance test. New senders (especially OpenTelemetry exporters) use it as a contract conformance check.

See Remote write feature page for operational guidance and Remote subsystem for the implementation.