Bucket replication

Asynchronous, per-bucket replication of object writes and deletes to one or more S3-compatible targets. Implemented in cmd/bucket-replication.go (~120 KB, the second largest file in the repo) on top of the bucket-targets registry in cmd/bucket-targets.go.

Purpose

• Mirror objects to a remote endpoint (another MinIO, AWS S3, or any S3-API-compatible store).
• Support active-active and active-passive setups.
• Track per-object replication state in xl.meta so retries are idempotent and cross-cluster reads stay consistent.

Layout

cmd/
├── bucket-replication.go              # Worker pool + replication loop
├── bucket-replication-handlers.go     # PUT/GET/DELETE bucket replication config
├── bucket-replication-utils.go        # Shared helpers
├── bucket-replication-utils_gen.go    # msgp codec
├── bucket-replication-stats.go        # Per-bucket stats
├── bucket-replication-metrics.go      # Aggregated metrics
├── bucket-replication-metrics_gen.go
├── bucket-targets.go                  # Target endpoint registry
internal/bucket/replication/
├── replication.go                     # Replication-config XML model
├── rule.go, filter.go, destination.go
├── status.go

Key abstractions

Symbol	File	What it is
BucketTarget	cmd/bucket-targets.go	Endpoint definition: URL, credentials, type.
BucketReplicationStats	cmd/bucket-replication-stats.go	Counters per bucket / target.
replicateObjectInfo	cmd/bucket-replication.go	One pending replication task.
ReplicationStatusType	internal/bucket/replication/status.go	PENDING, COMPLETED, FAILED, REPLICA.
replicationConfig	cmd/bucket-replication.go	Cached XML config + compiled rules.

How it works

graph LR
    PUT[PutObject completes] --> META[Mark xl.meta replication = PENDING]
    META --> ENQ[Enqueue replicateObjectInfo]
    ENQ --> WORKER[Worker pool]
    WORKER --> SDK[minio-go upload to target]
    SDK --> OK{ok?}
    OK -- yes --> DONE[Update xl.meta = COMPLETED]
    OK -- no --> RETRY[Update xl.meta = FAILED + scanner re-drive]

• The replication config is bucket-level XML following the AWS replication schema, parsed by internal/bucket/replication/.
• Each rule has filters (prefix, tags) and a destination (one or more BucketTarget references).
• A worker pool sized by MINIO_API_REPLICATION_WORKERS drains the queue. Stuck/failed entries are re-driven by the data scanner.
• Object versions and DELETE markers are replicated; replica metadata is preserved (X-Amz-Replication-Status, target-side ETag).
• Tagging changes, ACL changes, and lifecycle metadata changes are replicated separately as "metadata replication".

Existing-objects replication

When a rule is added, existing objects can be replicated by scanning the bucket and queuing each object as a replication task. This is the "existing object" workflow exercised by make test-replication.

Multipart replication

Multipart uploads are replicated as multipart on the target as well, preserving part boundaries. The worker tracks per-part state in xl.meta. See cmd/object-multipart-handlers.go for the source side.

Delete replication

Versioned delete markers replicate as delete markers; non-versioned deletes replicate as DELETE. Delete-marker proxying (returning the target's view to the source-side reader) is exercised by make test-delete-marker-proxying.

Bandwidth and proxying

Per-target bandwidth is tracked by internal/bucket/bandwidth/ and exposed through mc admin replicate. Read proxying (return the replica copy when the local copy hasn't yet caught up) is in cmd/bucket-replication.go and the related read paths in cmd/object-handlers.go.

Targets

cmd/bucket-targets.go is the source of truth for endpoint credentials. Targets persist into .minio.sys/buckets/<bucket>/bucket-targets.json and are propagated to peers via the peer REST. Target health is monitored separately so routing decisions can avoid a target known to be down.

Failure handling

• Transient errors (5xx, network) → exponential backoff in the worker.
• Permanent errors (403, 404) → marked FAILED, surfaced in metrics and to mc admin replicate status.
• Source-side restarts → on resume, the scanner re-drives all PENDING/FAILED entries.

Integration points

• Triggered by every PUT/POST/COPY/DELETE in cmd/object-handlers.go and cmd/object-multipart-handlers.go.
• Re-driven by cmd/data-scanner.go.
• Surfaced in metrics by cmd/metrics-v3-bucket-replication.go and cmd/metrics-v3-replication.go.
• Configured via the admin/management API in cmd/admin-handlers.go.

Test entry points

• make test-replication — 2-site and 3-site setups.
• make test-delete-replication — delete + delete-marker behaviour.
• make test-delete-marker-proxying — read-side proxying.
• make test-sio-error — SIO crypto error injection during replication.

The shell harness lives in docs/bucket/replication/.

Entry points for modification

• New per-rule semantic. Extend internal/bucket/replication/ and the compiled rules in cmd/bucket-replication.go.
• New target type. Subclass BucketTarget in cmd/bucket-targets.go and add the matching SDK in cmd/bucket-replication.go. (Today only S3-compatible targets are supported.)
• Worker tuning. MINIO_API_REPLICATION_WORKERS and friends in internal/config/api/.

See Site replication for the cluster-level mirror that builds on top of bucket replication.