Storage scrubber

The storage scrubber (storage_scrubber/) is an offline tool that scans cloud object storage and reports or deletes objects that no live tenant references. It exists because the pageserver cannot, on its own, reliably distinguish "this object belongs to a tenant I no longer host" from "this object is from a future generation I shouldn't touch."

Purpose

Layer files, index_part.json manifests, and partial WAL backups can become orphaned by:

• A tenant being deleted while one of its pageservers was offline.
• A failed shard split leaving behind objects under the old shard prefix.
• A generation bump after a pageserver crash, where the old generation's last few uploads landed in S3 after the controller had already moved on.
• Manual operator interventions during incident response.

The scrubber addresses this by walking the bucket tree, cross-referencing what it finds against the storage controller's authoritative tenant list, and emitting a report (or deletion plan) for everything it can't account for.

Modes of operation

storage_scrubber/README.md documents the modes the binary supports:

• scan-metadata — read every index_part.json and verify internal consistency. Detects tenants whose layer references are missing or whose generation numbers are inconsistent.
• scan-pageserver-metadata — same as above but scoped to one pageserver's prefix.
• tenant-snapshot — dump the full S3 view of one tenant for offline analysis.
• pageserver-physical-gc — list (or delete) layer files that no index_part.json references — i.e. orphaned objects.
• scan-safekeeper-metadata — analogous scrubber for safekeeper WAL backups.

A typical operations workflow:

1. Run scan-metadata regularly (daily or hourly cron) and alert on inconsistencies.
2. Run pageserver-physical-gc --mode=list to see how much could be reclaimed.
3. Run pageserver-physical-gc --mode=delete after operator review to actually delete.

Why it's offline

Putting the scrubber in the pageserver hot path would risk false-positive deletions during transient inconsistency (e.g. mid-split, mid-failover). Running it as a separate process with read-only S3 credentials by default lets operators verify proposed deletions before granting write access.

Implementation notes

The scrubber is built on top of the same libs/remote_storage and libs/pageserver_api crates as the pageserver. It re-uses:

• The same on-disk file naming convention to parse layer filenames.
• The same index_part.json schema.
• The same shard-key arithmetic to bucket layers per shard.

It talks to the storage controller via the storage_controller/client/ crate to get the list of currently-known tenants and shards.

Directory layout

storage_scrubber/
├── Cargo.toml
├── README.md          # operator-facing usage guide
└── src/               # ~6 KB Cargo.toml worth of code; smaller than the README

The crate is intentionally small; most of its work is delegated to the shared crates.

Key source files

File	Purpose
storage_scrubber/src/main.rs	Binary entry, subcommand dispatch.
storage_scrubber/src/scan_metadata.rs	Cross-references index_part.json against actual layer files.
storage_scrubber/src/pageserver_physical_gc.rs	Lists/deletes orphan layer files.
storage_scrubber/README.md	Operator usage.
libs/remote_storage/	The S3/Azure/GCS abstraction.
libs/pageserver_api/	Shared layer-naming and index_part.json schema.

See also

• Pageserver / Storage layers — the layout the scrubber audits.
• Storage controller — the source of truth for which tenants exist.
• Remote storage — the shared S3/GCS/Azure abstraction.