Cluster Mesh

Active contributors: aanm, giorio94, gandro, christarazi

Purpose

Cluster Mesh is Cilium's multi-cluster connectivity, identity, and policy feature. It lets pods in different Kubernetes clusters reach each other as if they were in one logical cluster, with consistent identity-based policy enforcement and global service discovery.

Each participating cluster runs a clustermesh-apiserver. Each agent in cluster A connects to cluster B's apiserver and vice versa.

Components

graph TB
    subgraph clusterA[Cluster A]
        A_API[kube-apiserver]
        A_CMA[clustermesh-apiserver A]
        A_etcd[(embedded etcd A)]
        A_Agents[Cilium agents A]
    end
    subgraph clusterB[Cluster B]
        B_API[kube-apiserver]
        B_CMA[clustermesh-apiserver B]
        B_etcd[(embedded etcd B)]
        B_Agents[Cilium agents B]
    end

    A_API --> A_CMA --> A_etcd
    B_API --> B_CMA --> B_etcd
    A_etcd <-.->|mTLS| B_Agents
    B_etcd <-.->|mTLS| A_Agents

What is propagated:

• Identities — every cluster exports its identities so policy can name remote workloads by labels.
• Services — services annotated for global access are exported and discoverable in every peer cluster.
• Endpoints — pod IP-to-identity bindings are exported for the ipcache.
• Nodes — node IPs and tunnel endpoints so the dataplane can route across clusters.

Directory layout

pkg/clustermesh/
├── manager.go                # cell that owns peer connections
├── common/                   # config and helpers
├── types/                    # cross-cluster type definitions
├── store/                    # shared-store consumers
├── endpointslicesync/        # MCS-API endpoint sync
├── mcsapi/                   # Multi-Cluster Services API integration
├── operator/                 # operator-side helpers
├── utils/                    # cluster name + cluster id helpers
└── ...

clustermesh-apiserver/
├── clustermesh/              # the export server
├── kvstoremesh/              # mesh proxy
├── etcdinit/                 # etcd bootstrap init container
└── ...

Cluster mesh life-cycle

1. Bootstrap. cilium clustermesh enable (the CLI command in cilium-cli/clustermesh/) generates mTLS materials, configures clustermesh-apiserver, and creates a Service (LoadBalancer or NodePort) that exposes the embedded etcd.
2. Peering. cilium clustermesh connect --context A --destination-context B exchanges certificates and mounts each cluster's etcd as a peer in the other.
3. Sync. Each agent watches every peer's etcd, populating its identity cache, ipcache, and service tables.
4. Policy. A CiliumNetworkPolicy can reference labels including io.cilium.k8s.policy.cluster=<name> to match workloads in a specific cluster.
5. Services. Annotating a service with service.cilium.io/global=true makes it visible everywhere; service.cilium.io/shared=false keeps it private even if global is set.

Failure modes

The mesh is designed to be partition-tolerant:

• If a peer cluster's apiserver is unreachable, agents continue to use their last-known state.
• Identity numbers can collide between clusters; Cluster Mesh disambiguates by composing identities with the cluster id (pkg/clustermesh/utils/).
• Services degrade gracefully: a global service backed only by failed remote backends is treated as unreachable rather than failing locally.

kvstoremesh

kvstoremesh (under clustermesh-apiserver/kvstoremesh/) is an optional optimisation for large meshes. Each cluster runs one kvstoremesh process that aggregates remote etcd connections; agents then read from the local kvstoremesh instead of opening N remote connections.

MCS-API

The Multi-Cluster Services API (pkg/clustermesh/mcsapi/, pkg/clustermesh/endpointslicesync/, clustermesh-apiserver/mcsapi-coredns-cfg/) is an alternative integration that exports services as ServiceImport / ServiceExport resources. CoreDNS configuration is rendered to resolve cross-cluster names like <service>.<ns>.svc.clusterset.local.

Integration points

• Identity allocator: identities exported via kvstore are merged into each agent's local cache.
• ipcache: remote endpoint IPs are added so the dataplane can resolve them to identities.
• Loadbalancer: global services are added to the local LB tables with backends pointing at remote pod IPs.
• Encryption: WireGuard / IPSec terminate at each node; the mesh transparently uses the node-to-node tunnels for cross-cluster traffic.

Entry points for modification

• New cross-cluster type: define a kvstore prefix, write a SharedStore consumer in pkg/clustermesh/store/, export it from clustermesh-apiserver/clustermesh/.
• New mesh-wide reconciliation: add a Hive cell under pkg/clustermesh/.
• MCS-API extensions: pkg/clustermesh/mcsapi/.

Key source files

File	Purpose
pkg/clustermesh/manager.go	Mesh manager cell.
pkg/clustermesh/types/cluster.go	Cluster ID and naming.
clustermesh-apiserver/clustermesh/	Export server.
clustermesh-apiserver/kvstoremesh/	Mesh proxy.
cilium-cli/clustermesh/	User-facing CLI.

See systems/kvstore.md and apps/clustermesh-apiserver.md.