Peer and storage RPC

In a distributed deployment the nodes need to talk to each other. MinIO uses three RPC layers, each tuned for a different traffic profile.

The three layers

Layer	Purpose	Files
Storage REST	Drive-level I/O between nodes that share an erasure set.	cmd/storage-rest-*.go
Peer REST	Cluster control plane: config reload, bucket metadata broadcast, metrics gather.	cmd/peer-rest-*.go, cmd/peer-s3-*.go
Grid	Binary multiplex protocol used for low-overhead control RPCs.	internal/grid/

The lock-rest server (cmd/lock-rest-server.go) is a fourth, narrower endpoint dedicated to DSync.

Storage REST

cmd/
├── storage-interface.go         # StorageAPI interface (drive contract)
├── storage-datatypes.go         # Wire types
├── storage-datatypes_gen.go     # msgp codec
├── storage-rest-common.go       # Path constants
├── storage-rest-common_gen.go
├── storage-rest-server.go       # /minio/storage/<set>/<drive>/* handlers
└── storage-rest-client.go       # Client side

The contract is StorageAPI (cmd/storage-interface.go). When an erasure set spans multiple nodes, the storage layer instantiates xlStorage for local drives and storageRESTClient for remote drives. They are interchangeable behind the interface.

Wire format: HTTP/1.1 with MessagePack request/response bodies. Streaming reads use chunked transfer; bitrot checksums are inlined. The sister file storage-datatypes_gen.go is the msgp codec.

Peer REST

cmd/
├── peer-rest-common.go          # Path constants
├── peer-rest-server.go          # Handlers
├── peer-rest-client.go          # Client
├── peer-s3-server.go            # S3-shaped peer endpoints
├── peer-s3-client.go
├── bootstrap-peer-server.go     # Lightweight bring-up endpoint
└── bootstrap-peer-server_gen.go

Used for:

• Bucket metadata broadcast (lifecycle, replication, encryption).
• IAM cache invalidation.
• Bandwidth, healing, and replication status fan-in for mc admin info.
• Cluster-wide trace stream.
• Peer health probing.

The "peer S3" endpoints (cmd/peer-s3-*.go) are S3-shaped variants used during bucket creation and deletion to coordinate listing and metadata writes across nodes.

Bootstrap peer

cmd/bootstrap-peer-server.go is a tiny endpoint exposed before the full server is up. It lets a node confirm it can reach a quorum of its peers during boot. Once the main server is up, this endpoint is shadowed by the regular peer REST.

Grid

internal/grid/
└── README.md   # Design + protocol description

A binary-multiplex protocol (one TCP connection carries many concurrent streams) optimised for many small messages. It's used for the control plane in newer code paths where the overhead of HTTP/1.1 framing is measurable.

The README in internal/grid/ describes the wire format in detail. The runtime exposes the grid over the same listener as the HTTP server with an upgrade handshake.

Authentication

Peer RPC traffic is signed using a cluster shared secret (the root credentials). All four layers verify signatures on every request. TLS is used end-to-end if MINIO_USE_TLS=on or certs are present.

Integration points

• Storage REST is used by cmd/erasure-server-pool.go whenever a target drive is on a remote node.
• Peer REST is called by cmd/iam.go (IAM reload), cmd/notification.go (config broadcast), and cmd/admin-handlers.go (cluster aggregations).
• Grid is used opportunistically by the lock layer and by newer admin endpoints.

Entry points for modification

• New drive call. Add a method to StorageAPI, implement it in xl-storage.go and storage-rest-client.go/storage-rest-server.go, add a path constant in storage-rest-common.go, regen msgp.
• New peer call. Same pattern in peer-rest-*.go.
• New grid endpoint. Implement under internal/grid/.