neondatabase/neon

Storage controller

The storage controller (storage_controller/) is the cluster manager for the storage tier. It maps tenants and shards onto pageservers, drives migrations, holds generation numbers that fence stale instances, and exposes a pageserver-compatible API so callers can talk to the cluster as if it were a single pageserver.

This page summarizes the design. The authoritative documentation in the repo is docs/storage_controller.md.

Purpose

A single pageserver process owns a set of tenant shards. The fleet has many pageservers. The storage controller answers questions like:

Which pageserver currently owns shard X of tenant Y?
When that pageserver dies, where does the shard re-attach?
How do I split a tenant from one shard into eight?
How do I bump generation numbers so the just-died pageserver can never accidentally write to S3 again after recovery?

It is not on the data path: a GetPage@LSN from a compute does not go through the storage controller. Instead, the controller publishes attachment information (often via the storage broker and the per-compute compute_hook) and the compute connects directly to the pageserver responsible for each shard.

Directory layout

storage_controller/
├── Cargo.toml
├── migrations/                # diesel SQL migrations for the metadata DB
├── client/                    # rust client crate
└── src/
    ├── lib.rs
    ├── main.rs                # binary entry; ~25 KB
    ├── service.rs             # the central state machine; ~444 KB (sic)
    ├── service/               # service.rs is split here for newer code
    ├── http.rs                # axum routes; ~91 KB
    ├── tenant_shard.rs        # per-shard intent/observed state; ~133 KB
    ├── reconciler.rs          # actuates intent → observed; ~52 KB
    ├── scheduler.rs           # placement algorithm; ~57 KB
    ├── persistence.rs         # diesel-based DB access; ~108 KB
    ├── persistence/
    ├── schema.rs              # diesel-generated DB schema
    ├── pageserver_client.rs   # client for talking to a pageserver
    ├── safekeeper_client.rs
    ├── safekeeper.rs          # SK-side reconciliation
    ├── compute_hook.rs        # notifies compute when its shard map changes
    ├── heartbeater.rs         # liveness checks against pageservers
    ├── leadership.rs          # PostgreSQL-advisory-lock-based HA
    ├── node.rs                # pageserver node abstraction
    ├── id_lock_map.rs         # fine-grained per-tenant locking
    ├── timeline_import.rs
    └── metrics.rs

control_plane/storcon_cli/    # CLI client (`storcon-cli`)

The biggest source files in the repo live here: service.rs at ~10.5k lines and tenant_shard.rs at ~3.2k lines.

Reconciliation model

The controller follows the standard "intent vs observed" reconciliation pattern:

graph LR
    Admin[Admin / API call] -->|set intent| Intent[Intent state<br/>tenant_shard.rs]
    Intent -->|diff vs| Observed[Observed state<br/>per-pageserver pull]
    Observed -->|drives| Reconciler[Reconciler<br/>reconciler.rs]
    Reconciler -->|HTTP calls| PS[Pageserver]
    Reconciler -->|notify| Compute[Compute via compute_hook]
    Reconciler -->|on success| Observed

Each tenant shard has:

An intent — where it should be attached, what its config should be.
An observed state — what each pageserver thinks it owns.
A reconciler background task that closes the gap.

reconciler.rs is the busy file: it issues the HTTP calls to attach/detach shards, updates remote storage index_part.json with new generation numbers, and notifies the compute via compute_hook.rs when its routing table changes.

Sharding

A tenant can be split from N shards into M (M = N × 2^k for some k). The split protocol:

Pause writes (block compaction, prevent compute writes from advancing past a chosen LSN — the controller coordinates with the compute).
Each existing shard partitions its layer files by the new shard map.
The new shards register with the controller, which assigns them to (possibly different) pageservers.
The compute is notified to use the new shard map; the old shards are marked detached.

The persistence DB tracks the split's progress so a controller restart mid-split picks up where it left off.

Generation numbers

Every attached shard has a monotonically increasing generation number. When the controller decides a pageserver is dead and re-attaches the shard elsewhere, it bumps the generation. Layer uploads to S3 include the generation in the object key prefix; the next pageserver that attaches uses the higher generation, so any zombie writes from the old (now-fenced) pageserver land in an "old generation" prefix that no live reader consults.

The pageserver fetches its current generation from the controller via the upcall API (/upcall/v1/) on attach. If a write fails because the generation has been bumped, the pageserver detaches itself.

This is why the controller's database must be durable — the docs warn against ephemeral disks here. Loss of generation numbers is loss of data safety.

API surface

docs/storage_controller.md enumerates four logically separate APIs, all served by http.rs:

Path	Purpose
`/v1/...`	Pageserver-compatible API for CRUD on tenants and timelines. Clients talk to the controller as if it were a pageserver.
`/control/v1/...`	Storage-controller-specific operations: register/manage pageservers, request shard splits, drain a node.
`/debug/v1/...`	Test-only or operator-only endpoints.
`/upcall/v1/...`	Called by pageservers: `re-attach`, `validate`. The generation-handshake mechanism.

Authentication uses the same JWT machinery as the pageserver, with scope pageserverapi.

Persistence

persistence.rs (~108 KB) wraps a Postgres database accessed via diesel. Migrations live in storage_controller/migrations/. The controller persists objects (tenants, nodes, shards) but not relationships — attachment state is kept in memory and rebuilt on startup by querying every registered pageserver.

To add a migration:

cargo install diesel_cli
diesel migration generate <name>
# edit the generated up.sql / down.sql
DATABASE_URL=postgresql://localhost:1235/storage_controller diesel migration run
git add storage_controller/migrations/<new> storage_controller/src/schema.rs

Migrations are baked into the binary and run automatically on startup.

Heartbeats and failover

heartbeater.rs periodically pings every registered pageserver. When a node misses too many heartbeats, the controller marks it offline and reconciliation reassigns its shards to other nodes. Generation numbers prevent the offline node, if it comes back, from interfering with the new attachments.

Secondary locations (warm standby copies of a tenant's data on a second pageserver) accelerate failover: the secondary already has the layer files cached, so the new attachment can serve traffic with minimal cold-start latency. Secondary locations are scheduled by scheduler.rs.

High availability of the controller itself

Multiple controller instances can run; one is leader at a time. leadership.rs uses Postgres advisory locks against the metadata DB to elect the leader. Followers are warm spares.

CLI: `storcon-cli`

control_plane/storcon_cli/ is a CLI that talks to the controller's HTTP API. Common commands:

storcon-cli tenant list
storcon-cli tenant describe <id>
storcon-cli tenant shard-split <id> --shard-count 8
storcon-cli node configure <id> --availability online

It's the day-to-day tool for storage operators.

Key source files

File	Purpose
`storage_controller/src/service.rs`	The central reconciliation state machine.
`storage_controller/src/tenant_shard.rs`	Per-shard intent/observed state.
`storage_controller/src/reconciler.rs`	Actuates intent → observed via HTTP calls.
`storage_controller/src/scheduler.rs`	Placement algorithm (where to put each shard).
`storage_controller/src/persistence.rs`	Diesel-based metadata DB access.
`storage_controller/src/http.rs`	All HTTP routes.
`storage_controller/src/compute_hook.rs`	Compute notification on shard moves.
`storage_controller/src/heartbeater.rs`	Pageserver liveness.
`storage_controller/src/leadership.rs`	Controller HA via Postgres advisory locks.
`storage_controller/migrations/`	Database schema migrations.
`docs/storage_controller.md`	Long-form documentation.