clustermesh-apiserver

Active contributors: aanm, giorio94, gandro, christarazi

Purpose

clustermesh-apiserver is the per-cluster control plane for Cluster Mesh. Each participating cluster runs one Deployment of this binary, which:

• Reads local Cilium identities, services, endpoints, and nodes from the kube-apiserver.
• Exports them into a small embedded etcd that is exposed (with mTLS) to peer clusters.
• Optionally runs kvstoremesh, which proxies cross-cluster reads to reduce per-agent connections.
• Provides health, readiness, and metrics endpoints for the mesh.

It also offers the MCS-API (mcsapi-coredns-cfg/) integration to advertise services as ServiceImports / ServiceExports for cross-cluster service discovery.

Directory layout

clustermesh-apiserver/
├── main.go
├── cmd/                       # cobra root + cell graph
├── clustermesh/               # the apiserver itself
├── clustermesh-dbg/           # debug CLI for the apiserver
├── kvstoremesh/               # in-process mesh proxy
├── kvstoremesh-dbg/           # debug CLI for kvstoremesh
├── etcdinit/                  # init container to set up embedded etcd
├── common/                    # shared helpers
├── health/                    # health endpoint
├── option/                    # config struct
├── syncstate/                 # sync coordination across components
├── metrics/                   # metrics
└── mcsapi-coredns-cfg/        # MCS-API CoreDNS config rendering

How it works

graph LR
    subgraph clusterA[Cluster A]
        A_API[kube-apiserver]
        A_CMA[clustermesh-apiserver]
        A_etcd[(embedded etcd)]
        A_Agent[cilium-agent]
    end
    subgraph clusterB[Cluster B]
        B_API[kube-apiserver]
        B_CMA[clustermesh-apiserver]
        B_etcd[(embedded etcd)]
        B_Agent[cilium-agent]
    end

    A_API -->|watch| A_CMA
    A_CMA -->|write identities, services| A_etcd
    A_etcd -.->|mTLS read| B_Agent
    B_API -->|watch| B_CMA
    B_CMA -->|write| B_etcd
    B_etcd -.->|mTLS read| A_Agent

Each agent in cluster A connects to cluster B's etcd (and vice versa). Identities and services are then merged into a shared identity space. The agent's pkg/clustermesh/ package handles the cross-cluster watchers and the merging logic.

kvstoremesh is an optional optimisation: instead of every agent connecting to every peer's etcd, agents connect to their local kvstoremesh which fans-in across peers. This caps the connection count at O(clusters) per cluster instead of O(agents × clusters).

Integration points

• Agents: read remote etcd via pkg/kvstore/ and merge in pkg/clustermesh/.
• CLI: cilium clustermesh connect (in cilium-cli/clustermesh/) generates the certificates and Service of type LoadBalancer that publishes the apiserver.
• CRDs: CiliumExternalWorkload, CiliumClusterConfig, plus the standard Identity/Endpoint resources are exported.
• Helm: all configuration flows through the chart under install/kubernetes/cilium/.

Entry points for modification

• New cross-cluster resources need:
  • A new export path in clustermesh-apiserver/clustermesh/
  • A consumer in pkg/clustermesh/
  • A schema in the kvstore prefix (pkg/clustermesh/types/)
• For tuning replication, look at pkg/kvstore/store/ (the generic shared-store abstraction).

Key source files

File	Purpose
clustermesh-apiserver/main.go	Entry point.
clustermesh-apiserver/cmd/	Cell graph.
clustermesh-apiserver/clustermesh/	Watch + export logic.
clustermesh-apiserver/kvstoremesh/	The mesh proxy.
pkg/clustermesh/	Agent-side consumer.
pkg/kvstore/etcd.go	etcd backend.

See features/cluster-mesh.md for the user-facing model.