Metacache and listing

ListObjectsV2 and ListObjectVersions against a deep bucket can be expensive — a MinIO cluster may have to walk hundreds of millions of xl.meta files. Metacache is a per-bucket streaming cache of list results that pays for itself across many concurrent listings.

Purpose

• Serve list-objects requests at near-constant cost regardless of bucket size.
• Stream listing across nodes so very large prefixes don't fill memory.
• Keep results consistent with versioning, lifecycle, and replication state.

Layout

cmd/
├── metacache.go              # Cache entry types, tracking
├── metacache-bucket.go       # Per-bucket metacache state
├── metacache-set.go          # Per-set scanning
├── metacache-server-pool.go  # Aggregation across pools
├── metacache-manager.go      # Lifecycle of metacache buckets
├── metacache-walk.go         # Filesystem walker (also used by data scanner)
├── metacache-entries.go      # Entry encoding/decoding
├── metacache-marker.go       # Continuation tokens
├── metacache-stream.go       # Streaming reader/writer over storage REST
└── bucket-listobjects-handlers.go # HTTP handlers

Key abstractions

Symbol	File	What it is
metacache	cmd/metacache.go	A single named cache (bucket + prefix + filter).
metaCacheEntry	cmd/metacache-entries.go	One listed entry (object or prefix).
metacacheBucket	cmd/metacache-bucket.go	All caches active in a bucket.
markerInfo	cmd/metacache-marker.go	A continuation token threading client requests.

How it works

graph LR
    REQ[ListObjectsV2] --> M[metacache match?]
    M -- hit --> SERV[Serve from existing cache]
    M -- miss --> NEW[Open new metacache]
    NEW --> WALK[Per-set parallel walk]
    WALK --> STREAM[metacacheBlock chunks]
    STREAM --> CLIENT[Stream entries to client]
    CLIENT --> NEXT[Next continuation token = block id]

One walker, many readers

When a list request comes in:

1. The handler hashes the request (bucket, prefix, delimiter, filters) and looks for an existing metacache.
2. If found, it streams from the cache's blocks, advancing the marker token.
3. If not, it kicks off a new walker. Walkers run per erasure set in parallel and append blocks to the cache.

A single walker can serve many concurrent list requests as long as their filters match. The cache TTL (default 5 minutes) limits how stale a result can be.

Cross-pool streaming

In a multi-pool deployment the per-pool walkers feed into cmd/metacache-server-pool.go, which merges entries by name to produce a single sorted stream. The merge is heap-based and respects delimiters.

Storage of cache blocks

Cache blocks live in .minio.sys/buckets/<bucket>/.metacache/. They are MessagePack-encoded entries (metacache-entries.go). When the cache expires or the bucket is empty, the manager (metacache-manager.go) deletes the directory.

Continuation tokens

The continuation token returned to the client encodes the cache id + block offset (cmd/metacache-marker.go). Clients can resume even if the original walker has finished.

Listing variants

• ListObjectsV1, ListObjectsV2, and ListObjectVersions all funnel through the same cache.
• mc ls uses V2.
• mc ls --versions uses ListObjectVersions.
• cmd/s3-zip-handlers.go uses listing to traverse zip indices stored on the server (github.com/minio/zipindex).

Integration points

• Calls into cmd/metacache-walk.go, which also powers the data scanner.
• Used by cmd/erasure-server-pool.go.ListObjects*.
• Bucket replication's "existing object" path uses listing under the hood.
• cmd/global-heal.go uses the walker for full scans.

Entry points for modification

• New filter dimension. Extend metacache.matches in cmd/metacache.go and the cache key.
• Tune TTL or block size. Constants live in cmd/metacache.go and cmd/metacache-bucket.go.
• New listing API. Reuse the manager and walker; add a handler in cmd/bucket-listobjects-handlers.go and an HTTP route in cmd/api-router.go.