istio/istio

istio-cni

Active contributors: zirain, ramaraochavali, hzxuzhonghu, keithmattix, stevenctl

Purpose

The istio-cni DaemonSet does two related but distinct jobs depending on which mesh mode is in use:

Sidecar mode: act as a CNI plugin that sets up iptables redirects in each application pod's network namespace at pod-create time, replacing the older privileged istio-init init container.
Ambient mode: act as a CNI plugin and as a node-local watcher/redirector. The plugin synchronously notifies a watch server when a pod with istio.io/dataplane-mode=ambient is scheduled. The watcher enters the pod's netns, programs iptables/nftables rules, and bridges traffic to the local ztunnel via Unix-domain sockets.

The DaemonSet packages two binaries:

cni/cmd/install-cni/ — runs as a long-lived process; installs the istio-cni binary into /opt/cni/bin/ and writes the CNI config in /etc/cni/net.d/. Watches the install paths and re-installs if anything is removed. In Ambient mode, also runs the node agent.
cni/cmd/istio-cni/ — the actual CNI plugin executable, invoked by kubelet/containerd via the CNI spec. Each invocation is a short-lived call.

The architecture document for Ambient redirection is architecture/ambient/ztunnel-cni-lifecycle.md.

Directory layout

cni/
├── README.md
├── cmd/
│   ├── install-cni/       # the long-running installer + node agent (Ambient)
│   └── istio-cni/         # the CNI plugin binary
├── deployments/           # Helm-ish DaemonSet templates (used by the istio-cni chart)
└── pkg/
    ├── plugin/            # CNI plugin handlers (CmdAdd / CmdDel / CmdCheck)
    ├── install/           # Install-watcher: copies the plugin binary, writes net.d config
    ├── nodeagent/         # The Ambient node agent: server, informers, ztunnel server
    ├── iptables/          # iptables rule construction (uses tools/istio-iptables)
    ├── nftables/          # nftables rule construction (newer kernels)
    ├── ipset/             # IPset helpers for ambient
    ├── repair/            # Sidecar-mode race-recovery for pod startup
    ├── trafficmanager/    # Ambient traffic redirect controller
    ├── pluginlistener/    # Local UDS the plugin uses to announce events to the agent
    ├── addressset/        # Address-set abstraction over ipset/nftables
    ├── log/               # Plugin-side logging that survives short-lived invocations
    ├── monitoring/        # Metrics
    ├── scopes/            # Log scopes
    ├── util/              # Shared helpers
    ├── config/            # CNI config types
    └── constants/         # Magic numbers shared across packages

Key abstractions

Symbol	File	Role
`Server`	`cni/pkg/nodeagent/server.go`	The Ambient node-agent main loop. Owns informers, traffic manager, ztunnel UDS.
`MeshDataplane`	`cni/pkg/nodeagent/meshdataplane_linux.go`	Implements per-pod redirect setup/teardown
`CniPluginServer`	`cni/pkg/nodeagent/cni-watcher.go`	UDS server that the plugin invocation talks to
`ZtunnelServer`	`cni/pkg/nodeagent/ztunnelserver.go`	Implements ZDS protocol toward the local ztunnel
`cmdAdd`, `cmdDel`, `cmdCheck`	`cni/pkg/plugin/plugin.go`	The CNI verbs
`IptablesConfigurator`	`cni/pkg/iptables/` (and `tools/istio-iptables/`)	Builds the iptables rule set
`Installer`	`cni/pkg/install/`	The long-running plugin-binary watcher
`PodCache`	`cni/pkg/nodeagent/pod_cache.go`	Tracks pod state across CNI calls and informer events

How it works

Sidecar mode

sequenceDiagram
    participant Kubelet
    participant CNI as istio-cni binary<br/>(in /opt/cni/bin)
    participant Pod as Pod netns
    Kubelet->>CNI: ADD <netns> <args>
    CNI->>CNI: parse annotations, decide redirect
    CNI->>Pod: nsenter; apply iptables rules
    Note over CNI,Pod: forwards 15001 / 15006 to istio-proxy
    CNI-->>Kubelet: success

The annotation-based selection API is documented in cni/README.md ("Selection API"). The plugin honors sidecar.istio.io/status, traffic.sidecar.istio.io/excludeOutboundPorts, etc. The repair controller (cni/pkg/repair/) handles a known race where a pod can start before its iptables rules are in place; it watches for unredirected pods and re-fixes them.

Ambient mode

sequenceDiagram
    participant Kubelet
    participant CNI as istio-cni plugin
    participant Watcher as nodeagent server
    participant Ztunnel as Local ztunnel pod

    Kubelet->>CNI: ADD <netns> <args>
    CNI->>Watcher: PluginAdd over UDS<br/>cni/pkg/pluginlistener
    Watcher->>Watcher: Pod cache update
    Watcher->>Pod netns: enter; install iptables<br/>+ create UDS socket pair
    Watcher->>Ztunnel: ZDS WorkloadAdded(uds_fd)<br/>cni/pkg/nodeagent/ztunnelserver
    Ztunnel-->>Watcher: ack
    Note over Pod netns,Ztunnel: One end of socket lives in pod ns<br/>other in ztunnel ns; iptables tunnels traffic

The node agent also runs a Kubernetes informer (cni/pkg/nodeagent/informers.go) that:

Watches namespace and pod labels for istio.io/dataplane-mode changes.
When a namespace is enrolled into ambient, finds pods on this node that already exist and need to be retroactively enrolled — it enters their netns and programs redirects without involving the CNI plugin (since CNI only fires on pod creation).
When a pod is unenrolled (label removed), tears down the redirects.

The ZDS protocol is defined in pkg/zdsapi/. Frames are length-prefixed protobuf over UDS with file-descriptor passing via SCM_RIGHTS — that's how the agent hands the workload's socket end to ztunnel without ztunnel needing CAP_SYS_ADMIN.

Traffic capture

For both modes, the actual iptables rules come from tools/istio-iptables/, the same Go library used by the legacy istio-init container. There is now also an nftables variant (cni/pkg/nftables/, tools/istio-nftables/) for kernels where iptables is being phased out.

Privileges

The DaemonSet runs privileged with hostNetwork: true and hostPID: true. At startup, it drops all Linux capabilities and re-adds only what it needs:

CAP_SYS_ADMIN — to enter pod netns.
CAP_NET_ADMIN — to program iptables/nftables.
CAP_NET_RAW — for IPv6 ICMP probes.

This is one of the strongest privilege levels in the install. Workloads remain unprivileged: pushing the privileged work into the node agent is the main security argument for the CNI architecture.

Integration points

Reads from: Kubernetes API (Pods, Namespaces) via the node agent's informers.
Writes to: pod network namespaces (iptables), local /opt/cni/bin/ and /etc/cni/net.d/ (binary + config), the local ztunnel pod (over UDS).
Interfaces with: the kubelet's CNI runtime (the plugin binary), the local ztunnel (ZDS), tools/istio-iptables and tools/istio-nftables (rule construction).
Health: a small HTTP server (cni/pkg/nodeagent/healthServer.go) exposes /healthz, used by the DaemonSet readiness probe.

Notable env vars

The most useful runtime knobs are in cni/README.md:

Env var	Default	Purpose
`HOST_PROBE_SNAT_IP`	`169.254.7.127`	SNAT IP for host probes; identifiable inside pod netns
`HOST_PROBE_SNAT_IPV6`	`fd16:9254:7127:1337:ffff:ffff:ffff:ffff`	IPv6 link-local equivalent
`AMBIENT_ENABLED`	(chart-driven)	Whether to start the Ambient node agent
`REPAIR_ENABLED`	(chart-driven)	Whether to run the sidecar repair controller
`LOG_LEVEL`	`info`	Plugin and agent log level

Entry points for modification

For new redirect behaviour, the right place is cni/pkg/iptables/ (or nftables/ if you support the alternative). Goldens for rule output live next door; regenerate with make refresh-goldens.
For changes to the Ambient pod lifecycle, the state machine lives in cni/pkg/nodeagent/server_linux.go + meshdataplane_linux.go. The pod cache (pod_cache.go) is the source of truth.
The CNI plugin itself (the short-lived binary) is in cni/pkg/plugin/. Be careful: a plugin invocation has no persistent state, so anything it needs must come from CNI args, env, or a UDS query.

Key source files

File	Purpose
`cni/cmd/istio-cni/main.go`	CNI plugin entry point
`cni/cmd/install-cni/main.go`	Installer + node-agent entry point
`cni/pkg/plugin/plugin.go`	CmdAdd / CmdDel / CmdCheck
`cni/pkg/install/install.go`	Plugin binary install + watch
`cni/pkg/nodeagent/server.go`	Node-agent server loop
`cni/pkg/nodeagent/server_linux.go`	Linux specifics
`cni/pkg/nodeagent/cni-watcher.go`	UDS server the plugin talks to
`cni/pkg/nodeagent/informers.go`	Pod / namespace informer logic
`cni/pkg/nodeagent/meshdataplane_linux.go`	Apply / remove redirect inside pod netns
`cni/pkg/nodeagent/ztunnelserver_linux.go`	ZDS server (talks to ztunnel)
`cni/pkg/nodeagent/pod_cache.go`	In-memory pod state
`cni/pkg/iptables/`	iptables rule builder
`tools/istio-iptables/`	The shared iptables library
`pkg/zdsapi/`	ZDS protobuf schema