Remote read

Remote read lets a Prometheus query an external store as if its data were local. The protocol is HTTP+protobuf; the local Prometheus issues ReadRequests and the remote endpoint streams back chunks.

Configuration

remote_read:
  - url: https://mimir.example.com/api/v1/read
    name: mimir-prod
    read_recent: true
    required_matchers:
      __name__: foo
    headers:
      X-Scope-OrgID: 'tenant-1'

Schema in config/config.go::RemoteReadConfig.

read_recent controls whether queries within --query.lookback-delta of "now" should be answered by the remote (set to false to keep recent queries local for latency).

How a query is routed

When a remote-read URL is configured, storage/remote.Storage.ApplyConfig registers a remote.Querier as a fanout secondary. PromQL queries that overlap the remote's time range fan out to both the local TSDB and the remote endpoint, and MergeSeriesSet deduplicates the results.

Protocols

• Buffered: Content-Type: application/x-protobuf — single response. Default.
• Streamed: Content-Type: application/x-streamed-protobuf;proto=prometheus.ChunkedReadResponse — interleaved chunks. Lower memory, latency-friendly. Server: storage/remote/read_handler.go::remoteReadStreamedXORChunks.

Server side

Local Prometheus exposes /api/v1/read (always; admin flags do not gate it). The handler is remote.NewReadHandler. It enforces:

• --storage.remote.read-sample-limit (default 5e7).
• --storage.remote.read-concurrent-limit (default 10).
• --storage.remote.read-max-bytes-in-frame (chunk framing budget).

Use cases

• A Prometheus that mostly stores recent data, falling back to long-term storage for historical queries.
• Migration: run two Prometheus instances side by side and query both transparently.
• Compatibility shim: query Cortex/Mimir/Thanos via the remote-read protocol from a Prometheus client that doesn't speak their native protocol.

Limitations

• Remote read is slower than native query paths in the remote backend; for production-scale, prefer the backend's native API where possible.
• Streaming read is not supported by every backend (Cortex/Mimir do; older targets may not).

Self-metrics

• prometheus_remote_storage_remote_read_queries_total{remote_name, code}
• prometheus_remote_storage_remote_read_query_duration_seconds
• prometheus_remote_storage_read_request_duration_seconds

Entry points for modification

• Streaming improvements: storage/remote/chunked.go and read_handler.go.
• New auth or transport: same hooks as Remote write.

See also: Remote write, Subsystems: Remote.