Cluster peering

Cluster peering connects two independent Consul clusters without a shared gossip pool, ACL system, or CA. It's the modern alternative to WAN federation and works between OSS and Enterprise, across cloud providers, and over public internet.

Purpose

WAN federation requires every server in every DC to know every other server, share gossip and a CA, and trust each other transitively. Peering instead establishes a one-to-one trust relationship between two clusters via mTLS-protected gRPC. Each side keeps its own root CA and exchanges trust bundles through a peering token.

Peering supports:

• Service discovery across the peer: services exported in cluster A become resolvable in cluster B as web.service.peer-a.consul.
• Mesh traffic across the peer: mesh gateways in each cluster forward mTLS connections to the right side.
• Per-service granularity: only the services that the operator explicitly exports are visible to the peer.
• Versioned independence: the two clusters can run different Consul versions and rotate ACLs independently.

Directory layout

agent/consul/
├── peering_backend.go            # Server-side gRPC implementation (16 KB)
├── peering_backend_ce.go
├── leader_peering.go             # Leader loop: replication, secret rotation (26 KB)
├── peering_endpoint.go           # HTTP endpoint forwarders
├── state/peering.go              # State store: peering, peering_secrets, peering_trust_bundles, exported_services (49 KB)
├── state/peering_ce.go
proto/private/pbpeering/         # Internal gRPC protocol (peering RPCs, replication)
agent/proxycfg/                   # Mesh-gateway snapshot work consumes peering data
agent/grpc-external/services/peerstream/  # The replication stream service
command/peering/                  # CLI: generate, establish, list, read, delete, exported-services

Key abstractions

Type	File	Purpose
Peering	agent/structs/peering.go	The persisted peering record (state, peer name, peer ID, trust bundle hash)
PeeringSecret	agent/structs/peering.go	Token, establishment, and stream secrets
PeeringTrustBundle	agent/structs/peering.go	The peer's CA roots
Backend (peering)	agent/consul/peering_backend.go	Implements the pbpeering.PeeringService server
Replication loop	agent/consul/leader_peering.go	Streams exported-services + roots to peers; refreshes secrets
pbpeering.PeerStreamService	agent/grpc-external/services/peerstream/	The bi-directional replication stream

Lifecycle

stateDiagram-v2
    [*] --> Pending: generate (cluster A)
    Pending --> Establishing: establish with token (cluster B)
    Establishing --> Active: mutual handshake + replication start
    Active --> Failing: stream errors / network partition
    Failing --> Active: auto-recovery
    Active --> Terminated: delete on either side
    Failing --> Terminated: delete on either side
    Terminated --> [*]

The PeeringState enum (agent/structs/peering.go) tracks INITIAL, PENDING, ESTABLISHING, ACTIVE, FAILING, TERMINATED, UNDEFINED, DELETING.

Peering establishment

sequenceDiagram
    participant OpA as Operator (A)
    participant ClusterA as Cluster A (server)
    participant OpB as Operator (B)
    participant ClusterB as Cluster B (server)

    OpA->>ClusterA: consul peering generate -name peer-b
    ClusterA->>ClusterA: state.PeeringWrite(initial, secrets generated)
    ClusterA-->>OpA: token (b64 with CA, address, secret)
    OpA-->>OpB: hand off token (out of band)
    OpB->>ClusterB: consul peering establish -name peer-a -peering-token=<token>
    ClusterB->>ClusterA: pbpeering.Establish (gRPC, mTLS via embedded bundle)
    ClusterA-->>ClusterB: ack
    ClusterB->>ClusterA: open PeerStream
    ClusterA-->>ClusterB: stream events (services, roots)
    ClusterB->>ClusterB: state.PeeringWrite(active)

The token is a base64 envelope containing the dialing endpoints, the peering UUID, the trust bundle, and the secret used by the establishing side to authenticate the first call. Token construction lives in agent/consul/peering_backend.go.

Replication

leader_peering.go runs one replication goroutine per active peering. It uses the peer-stream gRPC service (agent/grpc-external/services/peerstream/) to:

1. Stream exported services (the exported-services config entry filters which services are visible to the peer).
2. Stream CA roots of the local cluster.
3. Receive the peer's exported services and roots, persist them in peering_trust_bundles and the imported-services view.
4. Heartbeat to detect partitions and refresh the secret periodically.

The protocol is defined in proto/private/pbpeering/. Schema migrations are handled with backwards-compatible field additions.

Mesh integration

Once peering is active and services are exported, the proxy config manager (agent/proxycfg/) sees imported services and includes them in ConfigSnapshot.ConnectProxy.WatchedUpstreams and the mesh-gateway snapshot. The xDS server emits clusters and endpoints that route through mesh gateways to the peer:

• Cluster name encoding uses partition.namespace.peer-name (CE simplifies to peer-name).
• The mesh gateway in cluster A accepts mTLS traffic destined for cluster B services and tunnels them via SNI to cluster B's mesh gateway.

agent/xds/clusters.go and agent/xds/listeners.go have peer-aware paths; agent/proxycfg/mesh_gateway.go maintains the trust bundles per peer.

ACL handling

Peering uses dedicated tokens (peering "secret" tokens are server-only and rotated automatically). Operators control which services are exported via the exported-services config entry; ACL evaluation against the peer happens at export time.

Integration points

• Streaming subscribe: the peer-stream uses the same gRPC streaming machinery as the in-cluster subscribe path. State changes (config-entry updates, root rotation) immediately push to peers.
• Connect CA: peering trust bundles are persisted alongside the local CA roots so xDS can serve the union as Envoy SDS secrets.
• Operator UX: the consul peering CLI calls into the HTTP forwarders in agent/peering_endpoint.go which call the gRPC backend.
• V2 multicluster: internal/multicluster/ is the in-progress next-generation peering, layered on the v2 resource framework. Peering's underlying primitives are being generalized into resource-typed objects.

Entry points for modification

• Add a peering field: add to the protobuf in proto/private/pbpeering/, regenerate, update agent/structs/peering.go, the state store schema, and CLI rendering.
• Tune replication: see leader_peering.go::pollPeeringDialer and agent/grpc-external/services/peerstream/.
• Add a peer-aware endpoint: look at how agent/proxycfg/mesh_gateway.go consumes peering trust bundles for inspiration.
• Migrate to multicluster v2: the controller pattern under internal/multicluster/controllers/ is the destination.