kvstore

Active contributors: aanm, giorio94, gandro, christarazi

Purpose

The kvstore (pkg/kvstore/) is an etcd-backed shared key-value store. Originally the agent used it as the only source of truth for identities and services; today its primary remaining role is Cluster Mesh — propagating identities, services, endpoints, and node metadata between clusters.

Each cluster's clustermesh-apiserver runs an embedded etcd that other clusters' agents connect to (with mTLS). The kvstore package is the abstraction over this etcd connection.

Directory layout

pkg/kvstore/
├── kvstore.go         # the KVStore interface
├── etcd.go            # etcd implementation
├── client.go          # connection management
├── store/             # higher-level shared store with codecs
├── allocator/         # generic allocator built on kvstore (label-set -> id)
├── heartbeat/         # heartbeat key for liveness
└── tests/             # integration tests

Key abstractions

Type	File	Role
KVStore interface	pkg/kvstore/kvstore.go	The abstract contract: Get, Set, Update, Delete, ListPrefix, Watch.
etcdClient	pkg/kvstore/etcd.go	etcd v3 implementation with backoff, lease management, and metrics.
SharedStore	pkg/kvstore/store/	A typed view on a kvstore prefix that local consumers can Update and remote consumers can Watch.
Allocator	pkg/kvstore/allocator/	Generic allocator that maps a key (label set) to a numeric identity, used by the historical kvstore-based identity allocator.

How it is used today

graph LR
    ClusterA[Cluster A]
    ClusterB[Cluster B]
    A_etcd[(A etcd<br/>via clustermesh-apiserver)]
    B_etcd[(B etcd<br/>via clustermesh-apiserver)]
    A_Agent[A cilium-agents]
    B_Agent[B cilium-agents]

    ClusterA --> A_etcd
    ClusterB --> B_etcd
    A_etcd -.->|mTLS| B_Agent
    B_etcd -.->|mTLS| A_Agent

For Cluster Mesh, each agent in cluster A opens watches against cluster B's etcd (and vice versa). The watched prefixes carry:

• cilium/state/identities/v1/... — exported identities.
• cilium/state/services/v1/... — exported services.
• cilium/state/nodes/v1/... — node metadata.
• cilium/state/ip/v1/... — endpoint IP ↔ identity mappings.

Updates flow into local caches (pkg/clustermesh/) which merge them into the agent's identity, service, and ipcache views.

kvstoremesh

kvstoremesh (under clustermesh-apiserver/kvstoremesh/) is an optional optimisation: it terminates the per-cluster etcd connections at a single mesh proxy per cluster and re-publishes them locally to agents. This caps the number of cross-cluster TCP connections at O(clusters * N_clusters) instead of O(agents * N_clusters).

Backoff and resilience

The kvstore client has aggressive reconnection logic:

• Exponential backoff on connection failures.
• Heartbeat keys (pkg/kvstore/heartbeat/) detect stale connections.
• Lease management for ephemeral keys (e.g. node liveness).
• Metrics in pkg/metrics/ (cilium_kvstore_operations_total, cilium_kvstore_quorum_errors_total, ...).

Integration points

• Cluster Mesh: primary consumer — see features/cluster-mesh.md.
• Identity allocator (legacy): pkg/identity/cache/local_cache.go falls back to a kvstore allocator when CRD mode is disabled.
• External workloads: CiliumExternalWorkload resources are bootstrapped with kvstore-published state.

Entry points for modification

• New cross-cluster resource: define a new prefix and a SharedStore consumer on each side.
• New backend: implement KVStore interface (consul has been removed; etcd is the only supported backend today).
• TLS/auth changes: pkg/kvstore/etcd.go plus clustermesh-apiserver/etcdinit/.

Key source files

File	Purpose
pkg/kvstore/kvstore.go	KVStore interface.
pkg/kvstore/etcd.go	etcd implementation.
pkg/kvstore/store/	Generic shared-store abstraction.
pkg/kvstore/allocator/	kvstore-backed allocator.
Documentation/kvstore.rst	User-facing doc.