pkg/clustermesh and pkg/kvstore

Active contributors: aanm, giorio94, gandro, christarazi

Purpose

pkg/kvstore/ provides an etcd-backed KV abstraction. pkg/clustermesh/ consumes it to propagate identities, services, endpoints, and node metadata between Kubernetes clusters.

For the user-facing model see features/cluster-mesh.md. For the system view see systems/kvstore.md. This page is about the packages.

Layout

pkg/kvstore/
├── kvstore.go              # KVStore interface
├── etcd.go                 # etcd v3 backend
├── client.go               # connection management
├── store/                  # SharedStore: typed view on a kvstore prefix
├── allocator/              # KVStore-backed identity allocator (legacy)
├── heartbeat/              # liveness keys
└── ...

pkg/clustermesh/
├── manager.go              # cell that owns peer connections
├── types/                  # cluster ID, name encoding
├── store/                  # SharedStore consumers (identity, service, ipcache)
├── endpointslicesync/      # MCS-API EndpointSlice sync
├── mcsapi/                 # MCS-API integration
├── operator/               # operator-side helpers
├── utils/                  # helpers
├── common/                 # configuration, error types
└── ...

kvstore details

KVStore interface

type KVStore interface {
    Get(ctx, key) (val []byte, err error)
    Set(ctx, key, val) error
    Update(ctx, key, val, lease) error
    Delete(ctx, key) error
    ListPrefix(ctx, prefix) (KeyValuePairs, error)
    Watch(ctx, prefix) Watcher
    // ... lease/heartbeat helpers
}

The only production backend today is etcd. A consul backend existed historically and was removed.

SharedStore

pkg/kvstore/store/ builds a typed view on a kvstore prefix:

• The local side calls Update(item); the store serialises it into a key/value and pushes to etcd.
• Remote sides watch the prefix and receive Update / Delete callbacks.

This pattern is used for identities, services, endpoints, nodes — every cross-cluster resource in Cluster Mesh.

Cluster Mesh details

Manager

pkg/clustermesh/manager.go is a cell. Its responsibilities:

1. Watch the local /var/lib/cilium/clustermesh/<peer-name> config files (or the equivalent K8s Secret) to discover peers.
2. For each peer, open a kvstore client to its etcd endpoint with the configured CA + client cert.
3. Spawn SharedStore watchers for every cross-cluster type.
4. Inject received items into the agent's local caches (pkg/identity/cache/, pkg/ipcache/, pkg/loadbalancer/).

Cluster identity

Each cluster has:

• A cluster name (string, configured by cluster.name in Helm).
• A cluster ID (1-255). Cluster IDs are encoded into the high bits of NumericIdentity for cross-cluster identities so they don't collide with local ones.

pkg/clustermesh/types/cluster.go encodes/decodes these. pkg/clustermesh/utils/ provides helpers used elsewhere.

MCS-API

pkg/clustermesh/mcsapi/ and pkg/clustermesh/endpointslicesync/ implement the Multi-Cluster Services API, exposing services as ServiceImport / ServiceExport resources in addition to (or instead of) the kvstore-based mesh.

Integration points

• Identity allocator: kvstore-backed allocator (legacy) lives at pkg/kvstore/allocator/.
• Operator: Cluster Mesh operator helpers live at pkg/clustermesh/operator/.
• CLI: cilium clustermesh commands talk to the agent REST API for status and to the Helm chart for installation.

Entry points for modification

• New cross-cluster resource: define a kvstore prefix, write a SharedStore consumer in pkg/clustermesh/store/, register it from pkg/clustermesh/manager.go, and add an export path in clustermesh-apiserver/clustermesh/.
• New backend: implement KVStore. Be aware that backend changes are conservative — etcd is the only one that ships.
• TLS / authentication changes: pkg/kvstore/etcd.go and clustermesh-apiserver/etcdinit/.

Key source files

File	Purpose
pkg/kvstore/kvstore.go	KVStore interface.
pkg/kvstore/etcd.go	etcd backend.
pkg/kvstore/store/store.go	SharedStore.
pkg/clustermesh/manager.go	Mesh manager cell.
pkg/clustermesh/types/cluster.go	Cluster naming.
pkg/clustermesh/store/identity.go	Identity SharedStore consumer.