Site replication

Site replication keeps multiple MinIO clusters ("sites") in lock-step: every bucket, IAM user/group/policy, lifecycle rule, encryption setting, and replication target is kept consistent across sites. It's implemented in cmd/site-replication.go — at 189 KB, the largest production file in the repo.

Purpose

• Provide an "active-active across regions" deployment story without operator-managed scripts.
• Keep configuration uniform across sites: when a user is added on site A, they appear on B, C, ... within seconds.
• Replicate object data through the per-bucket replication paths once buckets exist on both sides.

Layout

cmd/
├── site-replication.go               # Top-level orchestration
├── site-replication-utils.go
├── site-replication-utils_gen.go
├── site-replication-metrics.go
├── admin-handlers-site-replication.go # /minio/admin/v3/site-replication/*
internal/config/...                   # consumed for IAM, lifecycle, encryption

Key abstractions

Symbol	File	What it is
SiteReplicationSys	cmd/site-replication.go	Cluster-wide coordinator (one per site).
madmin.SRClusterInfo	external madmin-go	Site definition wire type.
srStatusInfo	cmd/site-replication.go	Aggregate of bucket / IAM / policy state across sites.
peerSite	cmd/site-replication.go	Local view of a remote site (URL + creds + status).

Bootstrapping a new site

sequenceDiagram
    participant Op as Operator
    participant A as Site A
    participant B as Site B
    Op->>A: mc admin replicate add A B
    A->>B: GetClusterInfo (madmin)
    A->>A: Persist peerSite + creds
    A->>B: Push existing IAM, buckets, policies
    A->>B: Configure bucket replication targets
    Note over A,B: From this point on, both sites are equal peers.

The mc admin replicate add command lists the participating sites; the admin endpoint in cmd/admin-handlers-site-replication.go does the rest:

1. Pre-flight checks (versions, root credentials, existing bucket conflicts).
2. Exchange admin credentials between sites.
3. Push existing buckets and per-bucket configs (lifecycle, replication, encryption, tagging) to peers.
4. Push existing IAM users, groups, policies, and service accounts.
5. Configure bucket-replication targets on every site so subsequent writes flow.

Steady-state propagation

Every IAM mutation, bucket creation, lifecycle change, etc. that happens on any site is captured by the relevant subsystem and forwarded to peers. The hook points are:

• cmd/iam-store.go notifies SiteReplicationSys on user/group/policy/service-account changes.
• cmd/bucket-handlers.go and cmd/bucket-policy-handlers.go notify on bucket lifecycle / encryption / replication / tagging / quota changes.
• The peer REST server (cmd/peer-rest-server.go) exposes the receive side.

Failures are recorded in srStatusInfo; mc admin replicate status surfaces them. Re-syncing (mc admin replicate resync) replays IAM + bucket config from a chosen "source of truth" site.

Object data

Once the bucket is replicated to all sites, per-object replication uses the standard bucket replication machinery. Site replication configures every site's bucket-replication targets at bootstrap so writes converge on all sites.

Excluded entities

A few things deliberately don't replicate (current at the time of this snapshot of master):

• Server pool topology / decommission state.
• KMS keys (KES is expected to be the same KES across all sites).
• Local user passwords if the IDP is external (LDAP/OIDC) — the IDP is the source of truth.
• Per-site monitoring config.

Healing across sites

mc admin replicate edit and mc admin replicate add are the operator handles. The metrics in cmd/site-replication-metrics.go surface drift counts and last-success times per site.

Integration points

• IAM (cmd/iam.go, cmd/iam-store.go).
• Bucket replication (cmd/bucket-replication.go).
• Lifecycle, encryption, quota, tagging — all the subsystems that own per-bucket config push hooks here.
• The admin API (cmd/admin-handlers-site-replication.go).

Testing

• make test-site-replication-ldap (docs/site-replication/run-multi-site-ldap.sh)
• make test-site-replication-oidc (docs/site-replication/run-multi-site-oidc.sh)
• make test-site-replication-minio (docs/site-replication/run-multi-site-minio-idp.sh)

Entry points for modification

• Replicate a new entity type. Add a wire type to madmin-go, hook the source-side mutation site (cmd/iam.go, cmd/bucket-*.go), implement the receive side in cmd/admin-handlers-site-replication.go, plumb status into srStatusInfo.
• Tune resync. Resync is non-trivial; design notes are in the function-level comments inside cmd/site-replication.go.

See Replication for the per-bucket data path that site replication builds on.