Debugging

In-pod CLI: cilium-dbg

Every agent pod ships the cilium-dbg binary (built from cilium-dbg/). It is the primary tool for live debugging:

kubectl -n kube-system exec ds/cilium -- cilium status              # high-level health
kubectl -n kube-system exec ds/cilium -- cilium endpoint list       # local endpoints + identity
kubectl -n kube-system exec ds/cilium -- cilium policy get          # compiled policy
kubectl -n kube-system exec ds/cilium -- cilium identity list
kubectl -n kube-system exec ds/cilium -- cilium service list
kubectl -n kube-system exec ds/cilium -- cilium bpf ct list global
kubectl -n kube-system exec ds/cilium -- cilium bpf nat list
kubectl -n kube-system exec ds/cilium -- cilium bpf lb list
kubectl -n kube-system exec ds/cilium -- cilium bpf policy get <id>
kubectl -n kube-system exec ds/cilium -- cilium bpf ipcache list

Everything cilium-dbg shows comes from a BPF map or the agent's REST API (api/v1/). The CLI is generated from the OpenAPI spec under api/v1/openapi.yaml; new endpoints get a CLI subcommand for free.

Monitor and Hubble

The agent exposes a per-event monitor stream via a perf-event ring buffer (pkg/monitor/).

# Raw monitor output
kubectl -n kube-system exec ds/cilium -- cilium monitor

# Verbose (drop reasons, policy verdicts)
kubectl -n kube-system exec ds/cilium -- cilium monitor -v

# Hubble equivalent
hubble observe --pod default/foo
hubble observe --type policy-verdict --verdict DROPPED

Hubble parses the same monitor events into structured flows (pkg/hubble/parser/).

bugtool

bugtool/ collects a tarball of agent state for support cases:

kubectl -n kube-system exec ds/cilium -- cilium-bugtool
# Produces /tmp/cilium-bugtool-*.tar

It dumps every BPF map, every cmdref endpoint, log files, kernel info, and the Hubble flow buffer.

Common errors and where they originate

Symptom	Likely source
Pod stuck ContainerCreating with CNI errors	plugins/cilium-cni/, pkg/endpoint/restore.go, agent IPAM (pkg/ipam/)
Policy denies expected traffic	Compiled policy in pkg/policy/; check cilium endpoint config <id> and cilium policy get
Service traffic not load-balanced	LB BPF maps (pkg/maps/lbmap/), pkg/loadbalancer/, bpf/bpf_sock.c
DNS-based policy not working	pkg/fqdn/ and standalone-dns-proxy/
Cluster Mesh peers not connecting	clustermesh-apiserver/, pkg/clustermesh/ and the etcd in the remote cluster
Encryption broken	pkg/wireguard/ or pkg/datapath/linux/ipsec/ plus bpf/bpf_wireguard.c
eBPF verifier rejects program	bpf/complexity-tests/ and the BPF source in bpf/

Logs

The agent uses slog with structured fields. Log levels are configured by the --debug and --debug-verbose flags. Helpful filters:

kubectl -n kube-system logs ds/cilium | grep "level=ERROR"
kubectl -n kube-system logs ds/cilium | grep "subsys=daemon"
kubectl -n kube-system logs ds/cilium | grep -E "endpoint=([0-9]+)"

Subsystem names are constants in pkg/logging/logfields/ (e.g., subsys=policy, subsys=endpoint, subsys=k8s).

pprof and gops

The agent exposes pprof on localhost:6060 when --pprof is set, and a gops agent on localhost:9890 (port and address configurable):

go tool pprof http://localhost:6060/debug/pprof/heap
gops stack <pid>

See pkg/pprof/ and pkg/gops/.

BPF map inspection without cilium-dbg

bpftool works on every map but you need to know the key/value layout. Each map type's Go binding under pkg/maps/<map>/ exports the binary layout used to encode keys/values; the cilium bpf <map> subcommands wrap that and print human-readable output.