Healing

Healing is how MinIO repairs partial failures. There are several flavours, all of which converge on the same primitives in cmd/erasure-healing.go.

Purpose

Reconcile inconsistent state on disk: missing shards, bitrot-corrupted shards, missing object versions, missing buckets, brand-new disks that need to be repopulated, and operations that failed against a peer.

Flavours

Flavour	Trigger	File
Per-object heal-on-read	A read finds an inconsistency.	cmd/erasure-healing.go (HealObject)
Per-object heal-on-write	A write detects out-of-quorum metadata.	cmd/erasure-healing.go
Background heal	The data scanner emits hints; a worker picks them up.	cmd/global-heal.go, cmd/admin-heal-ops.go
New-disks heal	A drive shows up without format.json.	cmd/background-newdisks-heal-ops.go
Admin heal	mc admin heal issues an explicit job.	cmd/admin-heal-ops.go, cmd/admin-handlers.go
MRF (Most Recently Failed)	A peer was unreachable when an op was attempted.	cmd/mrf.go

Directory layout

cmd/
├── erasure-healing.go              # HealObject, HealBucket
├── erasure-healing-common.go
├── global-heal.go                  # Cluster-wide background heal worker
├── admin-heal-ops.go               # Heal job management (status, cancel)
├── background-heal-ops.go          # Heal task queue and dispatch
├── background-newdisks-heal-ops.go # New-drive bring-up
├── mrf.go                          # MRF queue + replay
├── healingmetric_string.go         # Metric enums
internal/config/heal/
└── heal.go                         # Config knobs (max_io, sleep, drive_workers)

Key abstractions

Symbol	File	What it is
healingTracker	cmd/background-newdisks-heal-ops.go	Per-drive heal progress (msgp-encoded *_gen.go).
healSequence	cmd/admin-heal-ops.go	Long-running heal jobs initiated by admin.
healSource, healEntry	cmd/global-heal.go	Hints from the data scanner.
partialOperation	cmd/mrf.go	One MRF entry.
madmin.HealOpts (external)	madmin-go	Knobs admin clients pass in.

How it works

Heal-on-read

graph LR
    GET[GetObject] --> META[Read xl.meta from each drive]
    META --> CHECK{All drives agree?}
    CHECK -- yes --> SERVE[Stream]
    CHECK -- no --> Q[Queue heal task]
    Q --> WORKER[Background heal worker]
    WORKER --> FIX[Re-encode missing shards / re-write meta]

The handler call HealObject is the entry; it picks the latest version present on a quorum of drives, re-uploads any missing shards, and rewrites xl.meta on out-of-date drives.

Background heal

cmd/global-heal.go runs a per-cluster goroutine that walks every bucket through the data scanner's tree and emits heal entries for objects whose meta is inconsistent. Workers (one per drive, count tunable via internal/config/heal/) consume the queue.

The job throttles itself based on cluster I/O. The relevant config keys (e.g. MINIO_HEAL_MAX_IO, MINIO_HEAL_DRIVE_WORKERS) live in internal/config/heal/heal.go.

New-disks heal

cmd/background-newdisks-heal-ops.go is started by cmd/server-main.go after the object layer is up. It:

1. Periodically scans local drives for missing format.json.
2. When it finds one, formats the drive (cmd/format-erasure.go).
3. Walks the other drives in the same set and rebuilds the new drive's shards.
4. Updates a healingTracker so mc admin info can show progress.

The msgp-generated background-newdisks-heal-ops_gen.go persists the tracker to disk so progress survives restarts.

MRF

When a write happens against a peer that is unreachable, the local server stages an MRF entry in cmd/mrf.go. A worker drains the queue and re-applies the operation once the peer returns. MRF state is persisted (msgp mrf_gen.go) so it survives restarts.

Admin heal

mc admin heal calls /minio/admin/v3/heal/.... The server dispatches via cmd/admin-heal-ops.go, returning a healSequence ID. Subsequent calls poll status. Cancellation is supported.

Integration points

• Reads triggered. cmd/erasure-decode.go and cmd/erasure-object.go enqueue heal tasks when shards are missing.
• Scanner triggered. cmd/data-scanner.go flags inconsistent objects.
• Quota / lifecycle. Healing must respect lifecycle (don't heal an object that's about to expire); see lifecycle interactions in cmd/global-heal.go.

Metrics

• minio_heal_* Prometheus families in cmd/metrics-v2.go.
• cmd/metrics-v3-cluster-erasure-set.go includes per-set heal stats.

Entry points for modification

• New heal kind. Add an entry to the heal task type, extend cmd/erasure-healing.go, surface it through admin-heal-ops.go.
• Throttling. Knobs live in internal/config/heal/. Adjust defaults there, document in Configuration.
• MRF retention. Tunable in cmd/mrf.go; default queue size is bounded.

See Erasure coding for the underlying read/write paths and Distributed locking for the concurrency model heal jobs work under.