Distributed object durability

How a single PUT survives node and disk failure once it lands in a MinIO cluster, and how the server keeps that promise over time.

What's happening

A successful PUT in a distributed MinIO cluster:

1. Picks an erasure set across nodes.
2. Erasure-encodes the payload into N data + M parity shards.
3. Writes each shard to a different drive (often a different node).
4. Writes xl.meta to every drive in the set.
5. Returns 200 only when a write-quorum of drives ack-ed.

After that, four background systems keep the object durable:

• The data scanner sweeps every object.
• Healing repairs missing shards / metadata.
• MRF replays operations that lost a peer at write time.
• Bitrot checking detects silent corruption on read.

The pieces

Concern	System
Encode / decode / shard layout	Erasure coding
Write quorum	Erasure coding, cmd/erasure-object.go
Cluster-wide ordering	Distributed locking
Per-node drive bring-up	Erasure coding, cmd/prepare-storage.go
Background reconciliation	Healing, Data scanner
Bitrot	Erasure coding, cmd/bitrot.go
Peer-to-peer transport	Peer and storage RPC

End-to-end

graph TD
    PUT[Client PUT] --> H[handler]
    H --> LK[NewNSLock]
    LK --> POOL[erasureServerPools.PutObject]
    POOL --> SET[erasureSets.PutObject]
    SET --> ENC[reedsolomon encode]
    ENC --> RPC[storage-rest write per drive]
    RPC --> XL[xl-storage]
    XL --> ACK[write quorum]
    ACK --> H
    SCAN[data scanner] -.->|repair hints| HEAL[heal]
    HEAL -.->|fix| XL
    MRF[MRF queue] -.->|replay on peer recovery| XL

Failure scenarios

Scenario	What happens
One drive in the set is offline at write time.	The remaining drives ack; MRF stages a replay against the offline drive.
Half the drives in a set are offline.	Read-quorum still possible; writes fail with ErrSlowDown.
One node in the set goes offline mid-write.	The op fails; client retries; MRF stages a replay.
One drive's xl.meta got corrupted.	Heal-on-read detects the inconsistency, re-writes the metadata.
One drive's shard payload got corrupted (bitrot).	The decoder rebuilds from parity; heal queues a repair.
A drive disappears (dead disk, replaced disk).	New-disks heal worker formats it and rebuilds shards from siblings.

Tuning knobs

• Set size. Default 16 drives. Larger sets give more parity but increase per-op fan-out.
• Parity ratio. Default EC:4 (4 parity drives in a 16-drive set). Configurable per storage class.
• Healing throughput. MINIO_HEAL_DRIVE_WORKERS, MINIO_HEAL_MAX_IO, MINIO_HEAL_SLEEP.
• Scanner cadence. MINIO_SCANNER_SPEED, MINIO_SCANNER_DELAY.

See Configuration for the full list.

Test entry points

• make verify-healing, make verify-healing-with-rewrite, make verify-healing-inconsistent-versions.
• make test-multipart (multipart quorum).
• make test-resiliency (full failure-injection harness via docs/resiliency/docker-compose.yaml).