Networking and routing

Active contributors: julianwiedmann, borkmann, brb, joestringer

Purpose

Cilium provides Layer 3 connectivity for pods. Each node runs the agent which:

• Allocates pod IPs from a pool (the IPAM mode).
• Programs routes (kernel routing table) and BPF programs (per-veth, per-host) so that packets reach the right pod.
• Optionally encapsulates traffic between nodes (overlay mode) or relies on the underlying network (native routing) to deliver pod IPs.

Routing modes

Two top-level options, controlled by routingMode in Helm:

Mode	Helm value	How packets cross nodes
Tunnel / overlay	tunnel (default for unfamiliar networks)	VXLAN or Geneve encapsulation handled by bpf/bpf_overlay.c. The host network only needs IP connectivity between nodes.
Native routing	native	The host network must be capable of routing pod CIDRs. Cilium installs routes; no encap. Often combined with BGP advertisement (pkg/bgp/).

Tunnel and native can be combined per-traffic class (e.g., overlay across regions, native within a zone).

CNI plugin

plugins/cilium-cni/ is the CNI binary that the kubelet calls when a pod is created. It:

1. Receives the network namespace path and pod metadata from the kubelet.
2. Asks the agent (over Unix socket) to create the endpoint.
3. The agent allocates an IP, creates a veth pair, sets up routes, and loads the per-pod BPF program.
4. Returns the pod's IP and DNS config to the kubelet.

The CNI binary is small; the heavy lifting happens in the agent. Endpoint state is persisted to disk (/var/run/cilium/state/<endpoint-id>/) so that the agent can restore endpoints after a restart without dropping running traffic.

IPAM modes

Pod IP allocation is configurable. Implementations live under pkg/ipam/:

Mode	Where pod IPs come from
kubernetes (default)	Kubernetes' built-in pod CIDR per node.
cluster-pool	A Cilium-managed cluster-wide pool, sliced into per-node pod CIDRs by the operator.
cluster-pool-v2beta	Newer variant with multiple pools and IPv6 support.
multi-pool	Per-namespace CiliumPodIPPool CRDs allow different pools per workload.
eni	AWS ENI mode. The operator allocates ENIs and IPs; the agent attaches them.
azure	Azure IPAM.
alibabacloud	Alibaba ENI.
crd	External controller manages the CRDs that publish IPs to the agent.

The agent's view of "which IPs are mine" comes from the CiliumNode CRD that the operator writes; the agent then claims an IP per endpoint creation request.

Per-pod plumbing

For each endpoint:

• A veth pair connects the pod netns to the host (pkg/datapath/connector/veth.go).
• The host-side end carries the per-endpoint BPF program (bpf/bpf_lxc.c compiled with the endpoint's ep_config.h).
• IPv4 and (optionally) IPv6 addresses are configured inside the netns.
• Static neigh entries are programmed for the gateway (the host).
• BPF maps are pinned: per-endpoint policy map, conntrack map (per-EP or global), NAT map, etc.

pkg/datapath/connector/ and pkg/datapath/link/ own the netlink work.

MTU

MTU is computed per-route based on routing mode + encapsulation overhead + encryption overhead. pkg/mtu/ resolves the host MTU and emits per-tunnel/per-encryption adjustments. The result is published in node_config.h so BPF programs know the right value to set on packet creation.

Multi-stack and multi-pool

• Dual-stack: controlled by ipam.ipv6.enabled; both IPv4 and IPv6 follow the same path.
• IPv6 only: supported with appropriate kernel and infrastructure.
• Multi-pool: CiliumPodIPPool CRDs (pkg/k8s/apis/cilium.io/v2alpha1/types_pod_ip_pool.go) let different namespaces draw from different IP ranges (useful for routing-policy compliance).

Integration points

• Datapath: all routing mode-specific behaviour is hidden behind compile-time flags in node_config.h.
• Operator: runs the IPAM controller in cloud modes (operator/pkg/ipam/).
• BGP: pkg/bgp/ advertises pod CIDRs in native mode.
• L2 announcer: pkg/l2announcer/ advertises VIPs via ARP in native mode.

Key source files

File	Purpose
plugins/cilium-cni/cmd/main.go	CNI binary entry.
pkg/datapath/connector/veth.go	veth setup.
pkg/ipam/manager.go	IPAM abstraction.
pkg/ipam/clusterpool/	cluster-pool mode.
pkg/aws/eni/	AWS ENI mode.
pkg/mtu/mtu.go	MTU resolver.
bpf/bpf_overlay.c	Tunnel encap/decap.
bpf/bpf_host.c	Per-host program.

See systems/datapath.md for the underlying eBPF mechanics.