neondatabase/neon

Storage broker

The storage broker is the small gRPC pub/sub mesh that lets pageservers and safekeepers find each other and exchange LSN updates without a central coordinator. It does not store data and does not participate in consensus — it's a fan-out broadcast service for cluster gossip.

Why it exists

In a Neon installation:

A safekeeper publishes its current flush_lsn for each timeline it serves.
A pageserver subscribes to flush_lsn updates so it knows whether it can pull more WAL.
Pageservers can also publish their own last_record_lsn and disk_consistent_lsn so other components (e.g. the storage controller) can watch lag.

Doing this with point-to-point connections doesn't scale — each pageserver would need a connection to every safekeeper. The broker centralizes the fan-out.

What it actually does

storage_broker/src/ is a tonic-based gRPC service. Clients send messages of two kinds:

Publish — "this is my current state for (tenant_id, timeline_id, role)." The broker keeps the most recent message per key and forwards it to subscribers.
Subscribe — "send me every update matching this filter." The broker pushes matching messages as they arrive.

Messages are tiny (a handful of integers and IDs) and frequent. The broker keeps state only in memory; restarts are tolerable because clients re-publish on reconnect.

Directory layout

storage_broker/
├── Cargo.toml
├── build.rs                  # invokes tonic-build over proto/
├── proto/
│   └── broker.proto          # service + message definitions
├── src/
│   ├── lib.rs                # crate root
│   └── bin/storage_broker.rs # binary entry point
└── benches/                  # broadcast latency benchmark

The protocol is defined in storage_broker/proto/broker.proto. The Rust code is short — most of the substance is in tonic boilerplate plus a small in-memory subscription manager.

How it fits in

graph TD
    SK1[safekeeper 1] -->|publish flush_lsn| Broker[storage_broker]
    SK2[safekeeper 2] -->|publish flush_lsn| Broker
    SK3[safekeeper 3] -->|publish flush_lsn| Broker
    Broker -->|subscribe (tenant_id,timeline_id)| PS1[pageserver 1]
    Broker -->|subscribe| PS2[pageserver 2]
    PS1 -->|publish last_record_lsn| Broker
    Storcon[storage_controller] -.->|subscribes for monitoring| Broker

A pageserver typically subscribes to all timelines it has attached. When it sees a safekeeper's flush_lsn jump, it knows there is new WAL to fetch and starts (or resumes) a libpq replication stream against that safekeeper. Conversely, when the broker reports that safekeeper has the WAL the pageserver needs, the storage controller can use this to schedule reconciliation.

Failure model

The broker is intentionally stateless. If it crashes, all clients reconnect, re-publish their current state, and re-establish subscriptions. The price is a brief blackout — during the gap, pageservers don't get fresh WAL position updates. The fallback is "poll the safekeeper directly" once the existing connection times out.

In production deployments, multiple broker instances run behind a load balancer for availability.

Authentication

The broker uses the same JWT scheme as the pageserver and safekeeper. See libs/utils/src/auth.rs for the verification code.

Key source files

File	Purpose
`storage_broker/proto/broker.proto`	gRPC service + message definitions.
`storage_broker/src/bin/storage_broker.rs`	Server binary, in-memory state, subscriber registry.
`storage_broker/src/lib.rs`	Library exports for clients.
`pageserver/src/controller_upcall_client.rs`	Pageserver-side broker client.
`safekeeper/src/broker.rs`	Safekeeper-side broker client.
`docs/storage_broker.md`	One-paragraph summary in repo docs.