Erasure coding

MinIO's storage backend is a custom erasure-coded layer that splits each object into data + parity shards across drives. It tolerates the loss of half its drives in a set, supports versioning, encryption, and inline tiny objects, and persists everything through a single sidecar metadata file: xl.meta.

Purpose

Provide a cluster-wide, durable, object-addressed storage layer that scales by adding pools of nodes/drives, repairs itself on the fly, and presents a stable interface (ObjectLayer) to the rest of the server.

Hierarchy

graph TD
    POOL[erasureServerPools] -->|N pools| ZN[erasureSets per pool]
    ZN -->|M sets| ESET[erasureSets — group of drives in one EC domain]
    ESET -->|K drives| DRV[xl-storage / storageRESTClient]
    DRV --> META[xl.meta]
    DRV --> SHARDS[part shards]

Terminology:

• Pool. A homogeneous group of nodes added together. Defined per command-line argument in cmd/erasure-server-pool.go.
• Erasure set. The smallest erasure-coding domain; usually 16 drives. Defined in cmd/erasure-sets.go.
• Drive. One filesystem mount. Local drives use cmd/xl-storage.go; remote drives use cmd/storage-rest-client.go.

Directory layout

cmd/
├── object-api-interface.go        # ObjectLayer
├── erasure-server-pool.go         # Top-level implementation of ObjectLayer
├── erasure-server-pool-decom.go   # Decommission a pool's contents into others
├── erasure-server-pool-rebalance.go # Rebalance objects across pools
├── erasure-sets.go                # Pool of sets within one pool
├── erasure.go                     # erasureObjects = one set
├── erasure-object.go              # Per-object PUT/GET on a set
├── erasure-multipart.go           # Multipart upload semantics on a set
├── erasure-coding.go              # Wrapper around klauspost/reedsolomon
├── erasure-encode.go              # Encode pipeline
├── erasure-decode.go              # Decode pipeline
├── erasure-healing.go             # Per-object healing
├── erasure-healing-common.go
├── erasure-metadata.go            # Validate / merge xl.meta across drives
├── erasure-metadata-utils.go
├── erasure-utils.go
├── erasure-common.go
├── format-erasure.go              # format.json on each drive
├── format-meta.go                 # Common format helpers
├── prepare-storage.go             # Drive bring-up + quorum wait
├── xl-storage.go                  # Local drive implementation
├── xl-storage-disk-id-check.go    # Dedupe drive identity guard
├── xl-storage-format-v2.go        # Current xl.meta format
├── xl-storage-format-v2-legacy.go # Migration helpers
├── xl-storage-format-v1.go        # Original metadata format
├── xl-storage-meta-inline.go      # Inline tiny objects in xl.meta
├── xl-storage-free-version.go     # Soft-deleted version handling
├── bitrot.go, bitrot-streaming.go, bitrot-whole.go # HighwayHash bitrot
├── storage-interface.go           # StorageAPI interface
└── storage-datatypes.go, storage-rest-*.go # Inter-node storage RPC

Key abstractions

Symbol	File	What it is
ObjectLayer	cmd/object-api-interface.go	The interface every backend implements.
erasureServerPools	cmd/erasure-server-pool.go	Top-level ObjectLayer; owns N pools.
erasureSets	cmd/erasure-sets.go	One pool's collection of sets.
erasureObjects	cmd/erasure.go	One erasure set; the unit of EC math.
StorageAPI	cmd/storage-interface.go	Drive-level interface (xl-storage, REST client, naughtyDisk).
xlStorage	cmd/xl-storage.go	Local on-disk driver.
storageRESTClient	cmd/storage-rest-client.go	Remote drive driver over HTTP.
xlMetaV2	cmd/xl-storage-format-v2.go	The current MessagePack metadata.
formatErasureV3	cmd/format-erasure.go	Drive identity (format.json).
Erasure	cmd/erasure-coding.go	Wraps reedsolomon for encode/decode.

How it works

Drive bring-up

cmd/prepare-storage.go reads the endpoints, loads each drive's format.json, waits for read-quorum, and produces a slice of StorageAPI handles. New drives without a format.json are detected and queued for healing by cmd/background-newdisks-heal-ops.go.

Object PUT path

graph LR
    H[PutObject handler] --> POOL[erasureServerPools.PutObject]
    POOL --> SETSEL{Pick pool/set}
    SETSEL --> SET[erasureObjects.PutObject]
    SET --> ENC[erasureEncode]
    ENC --> RS[reedsolomon Encode]
    RS --> SHARDS[N data + M parity shards]
    SHARDS --> WRITE[Write each shard to its drive]
    WRITE --> META[Write xl.meta to all drives]

• The pool/set selector is biased toward the pool with the most free space (cmd/erasure-server-pool.go).
• Each shard goes to its own drive; the encoder is cmd/erasure-encode.go.
• A HighwayHash checksum per shard is written into xl.meta for bitrot verification.
• A successful PUT requires write-quorum (Math.Ceil((N+M)/2) + 1).

Object GET path

graph LR
    G[GetObject handler] --> POOL[erasureServerPools.GetObject]
    POOL --> SET[erasureObjects.GetObject]
    SET --> READMETA[Read xl.meta from drives]
    READMETA --> PICK[Pick best version + parts]
    PICK --> READSH[Read N shards in parallel]
    READSH --> DEC[reedsolomon Reconstruct if needed]
    DEC --> CLI[Stream to client]

If any shards are missing or fail bitrot verification, the decoder reads from parity shards and the heal-on-read path queues a repair job. This is implemented in cmd/erasure-decode.go and the healing wrappers in cmd/erasure-healing.go.

xl.meta

xl.meta is a single MessagePack file per object on each drive. It carries:

• All versions of the object (current + non-current + free).
• Each version's parts, their sizes, and HighwayHash checksums.
• Inline data for tiny objects (small files are stored entirely inside xl.meta to save the cost of a separate shard file — cmd/xl-storage-meta-inline.go).
• Encryption metadata (envelope key, KMS context).
• Replication state.
• Lock metadata.

The file is generated/decoded by msgp; see cmd/xl-storage-format-v2_gen.go. Migration from v1 → v2 is done in cmd/xl-storage-format-v2-legacy.go.

Free versions

When a versioned object is deleted, the version is converted to a "free version" instead of being immediately purged. The data scanner and lifecycle subsystem decide when free versions become reclaimable. See cmd/xl-storage-free-version.go.

Bitrot

Every shard write computes a HighwayHash over the bytes; the hash lives in xl.meta. Reads recompute and compare. There are three variants:

• cmd/bitrot.go — common types.
• cmd/bitrot-streaming.go — streaming reads with progressive checksum.
• cmd/bitrot-whole.go — small whole-shard checksum (used for inline data and meta).

Decommission and rebalance

• Decom (cmd/erasure-server-pool-decom.go) drains a pool: every object is re-uploaded to a different pool, then the pool is removed from the topology.
• Rebalance (cmd/erasure-server-pool-rebalance.go) spreads existing objects more evenly across pools after expansion.

Both operations are driven by admin endpoints (cmd/admin-handlers-pools.go) and exposed to mc admin pool / mc admin rebalance.

Integration points

• Implements ObjectLayer consumed by every HTTP handler.
• Uses cmd/namespace-lock.go for distributed locks.
• Drives cmd/data-scanner.go and cmd/global-heal.go for background work.
• Emits data-usage-cache updates (cmd/data-usage-cache.go).
• Calls into internal/crypto/ for SSE encryption-on-write.

Entry points for modification

• Add a new drive call. Add the method to StorageAPI (storage-interface.go), implement in xl-storage.go and the REST client, register the route in storage-rest-server.go. Tests in xl-storage_test.go and storage-rest_test.go.
• Change xl.meta. Bump the version in xl-storage-format-v2.go, add a migration in xl-storage-format-v2-legacy.go, regenerate msgp.
• Tune the encoder. erasure-coding.go is the only place reedsolomon is constructed; performance experiments belong here.
• Add a heal mode. Extend the heal task type in erasure-healing.go and wire the new mode through global-heal.go and admin-heal-ops.go.

See Healing for how healing works on top of this and Distributed locking for the concurrency model.