Background

Design rationale, recurring pitfalls, and migration context that an engineer should know before making invasive changes.

Design decisions

Why eBPF rather than iptables / nftables

Cilium's founding bet was that eBPF would let the dataplane scale beyond what iptables can. iptables suffers under cluster sizes where every Service produces dozens of rules; nftables is better but still struggles with cross-cutting features like identity-based policy. eBPF lets Cilium implement policy lookup as a single hash table read regardless of cluster size.

The trade-off is verifier complexity, kernel-version coupling, and the need for clang/LLVM at runtime. The agent ships a clang/LLVM toolchain because BPF programs are compiled per-endpoint at start.

Why identities, not IPs

IPs are ephemeral and reused. Encoding "what is allowed to talk to what" in IPs means rebuilding rules every pod restart. Cilium derives a stable identity from each pod's labels and lets policy be identity_X -> identity_Y. The dataplane then resolves source/destination IPs to identities through the ipcache map.

This decoupling also makes Cluster Mesh feasible: identities can be exported across clusters without leaking per-pod IPs.

Why Hive

Before Hive, the agent's daemon.NewDaemon() constructor was thousands of lines, with manual ordering of subsystem initialization. New features had to find a place in this giant function and respect the order or break startup. Hive replaces this with a declarative cell graph: each subsystem says "I need X and Y, I provide Z, my OnStart does ...". Hive resolves the order automatically.

The migration is incremental. New code is expected to be cells; old code is converted opportunistically.

Why StateDB

Shared mutable state (routes, devices, services, neigh entries) was previously kept in package-local maps with mutexes plus subscriber lists. Each new feature meant a new map and a new ad-hoc notification mechanism. StateDB collapses this into a transactional table abstraction with built-in change watchers and reconcilers. It also gives consistent point-in-time reads, which matters for cross-table updates.

Why Envoy for L7

Cilium had a custom Go proxy in v1.0; it was replaced by Envoy because Envoy's filter ecosystem (TLS termination, observability, gRPC, Kafka, HTTP/2) is far broader. Cilium contributed a custom cilium.l7policy filter so that identity-aware decisions can be made inside the existing Envoy filter chain.

The cost is operational: Envoy is heavy and has its own learning curve. The single-Envoy-per-node model (rather than a sidecar per pod) keeps the cost contained.

Pitfalls

Verifier complexity

The Linux BPF verifier limits program size and complexity. Adding what looks like simple branches in bpf/ files can suddenly cause programs to fail to load on older kernels. The bpf/complexity-tests/ suite catches the regressions CI cares about. Check there before adding inline code paths.

Map lifetimes and pinning

Maps are pinned to bpffs (/sys/fs/bpf/cilium/...) so they survive agent restarts. Renaming or restructuring a map without a migration in pkg/maps/<name>/migrate.go will break upgrade. The agent does best-effort migration but cannot recover from drastic schema changes.

Endpoint regeneration storms

A single label change on a popular pod can trigger policy recomputation for every endpoint that selectors that label set. The build queue (pkg/endpoint/endpoint_build_queue.go) caps parallelism, but expensive policies plus large clusters can produce regeneration storms. Check cilium_endpoint_regenerations_total and cilium_endpoint_state distributions when investigating perf issues.

Identity GC timing

The operator garbage-collects unused identities. If GC is too aggressive, recreated pods can briefly miss policy. If GC is too lax, identities accumulate. The default values err on the side of caution; do not lower them without good reason.

Cluster Mesh consistency

Cluster Mesh depends on every agent watching every peer cluster's etcd. Network partitions cause stale state on either side. The agent caches identity and service data so traffic continues to work but may use slightly outdated information. Don't rely on Cluster Mesh as a strongly consistent datastore — it is eventually consistent.

Helm values vs ConfigMap drift

The chart renders Helm values into a ConfigMap that the agent reads. Manual edits to the ConfigMap will be reverted on the next Helm reconcile. Always change Helm values, not the ConfigMap directly.

CIDR vs Identity policy

toCIDR and fromCIDR in policy create synthetic identities for matching IPs. This works for static CIDR ranges but does not adapt to cluster topology — for in-cluster traffic, prefer identity-based selectors.

Migration history

• v1.6 (2019) — Hubble extracted from the agent's monitor.
• v1.7 – v1.10 — kube-proxy replacement, Maglev, DSR.
• v1.11 — CES (CiliumEndpointSlice) for scale.
• v1.12 — bandwidth manager (EDT), wireguard preview.
• v1.13 — start of the Hive migration; identity allocator unified.
• v1.14 – v1.16 — Gateway API, mutual auth, BGPv2 design.
• v1.17 – v1.19 — Service mesh hardening, standalone DNS proxy, StateDB conversions, ztunnel scaffolding.

For commit-level history see lore.md.

Reading further

• systems/control-plane.md — current Hive architecture.
• systems/datapath.md — kernel-side dataplane.
• packages/policy.md — policy compilation.
• Documentation/internals/ — deeper technical notes from the maintainers.