Data scanner

The data scanner is the background sweep that walks every object in the cluster every few hours. It's how usage caches stay current, lifecycle rules fire, replication queues get re-driven, and healing finds work to do.

Purpose

• Maintain data-usage-cache so quotas, ILM, and metrics are correct.
• Apply lifecycle expirations and transitions.
• Surface inconsistencies for the heal subsystem.
• Re-drive failed replication entries.

Layout

cmd/
├── data-scanner.go            # Main loop + per-bucket scan
├── data-scanner_test.go
├── data-scanner-metric.go     # Metrics for scanner state
├── scannermetric_string.go
├── data-usage.go              # Aggregation across buckets
├── data-usage-cache.go        # Per-bucket persistent cache
├── data-usage-utils.go
├── data-usage-cache_gen.go    # msgp-generated codec
└── metacache-walk.go          # The actual filesystem walk used by the scanner

Key abstractions

Symbol	File	What it is
folderScanner	cmd/data-scanner.go	Recursive walker over one bucket prefix tree.
dataUsageCache	cmd/data-usage-cache.go	Per-bucket size / count / lifecycle state cache.
scannerStats	cmd/data-scanner-metric.go	Counters surfaced via metrics.
walkVersionsFn	cmd/metacache-walk.go	Lower-level walk used both by the scanner and listing.

How it works

graph TD
    LOOP[scannerCycle goroutine] --> PICK[Pick bucket]
    PICK --> WALK[metacache-walk over set]
    WALK --> EVAL[Per-object evaluation]
    EVAL --> ILM{lifecycle?}
    EVAL --> HEAL{healing hint?}
    EVAL --> REPL{replication backlog?}
    EVAL --> USAGE[Update dataUsageCache]
    ILM --> EXPIRE[Queue expiration / transition]
    HEAL --> Q[Queue heal task]
    REPL --> RREPL[Re-drive replication]
    LOOP --> PERSIST[Persist cache + metrics]

The cycle period and worker count are governed by internal/config/scanner/. Default behaviour is throttled — the scanner sleeps between objects to keep impact on user traffic small.

Data usage cache

dataUsageCache is a tree of folders + sizes + counts persisted into .minio.sys/buckets/<bucket>/usage-cache.bin. It's the source of:

• mc admin info cluster usage figures.
• Bucket quota enforcement (internal/config/heal/, cmd/bucket-quota.go).
• Prometheus metrics under minio_cluster_usage_*.

The cache uses MessagePack (data-usage-cache_gen.go) for compact, fast persistence.

Lifecycle integration

The scanner is the trigger for ILM. While walking, it loads the bucket's lifecycle config (internal/bucket/lifecycle/) and decides for each object version whether it should be expired, transitioned, or left alone. Decisions go to the worker pool in cmd/bucket-lifecycle.go.

Replication integration

If a bucket is configured for replication and the scanner finds objects whose replication status is PENDING or FAILED, it re-queues them on the replication worker (cmd/bucket-replication.go).

Heal integration

Inconsistencies (missing version on a drive, bitrot-flagged shards) are emitted as heal hints; the background heal worker (cmd/global-heal.go) consumes them.

Throttling

The scanner respects:

• MINIO_SCANNER_SPEED (slow/default/fast).
• MINIO_SCANNER_DELAY between objects.
• Cluster-wide I/O caps via internal/config/heal/.

These defaults are intentionally conservative — the scanner is supposed to be invisible.

Integration points

• Reads through metacache-walk.go.
• Writes to dataUsageCache.
• Calls into internal/bucket/lifecycle/ for ILM decisions.
• Emits heal entries to cmd/global-heal.go.
• Re-drives replication via cmd/bucket-replication.go.

Metrics

• minio_scanner_* family in cmd/metrics-v2.go.
• cmd/metrics-v3-scanner.go for the v3 endpoints.

Entry points for modification

• New per-object decision. Extend the inner loop in cmd/data-scanner.go. Keep it side-effect-free; emit work to the relevant worker queue.
• Cache schema. Bump the version in cmd/data-usage-cache.go, regen msgp, add a migration.
• Pace. Adjust defaults in internal/config/scanner/.

See Lifecycle and tiering, Healing, and Replication for downstream consumers.