Debugging

How to investigate problems both at development time (in unit tests) and against a running cluster.

Logging

Istio's log facade is pkg/log/, a structured layer over zap. The unit you tune is a scope: every subsystem registers a named scope (e.g. pilot, ads, model, kube, ca, injector, serviceregistry, wasm, validation).

Adjusting log levels

For istiod (and any binary using pkg/log):

# At process start
istiod --log_output_level=ads:debug,model:debug

# At runtime, via the ControlZ HTTP UI on port 9876 by default
curl localhost:9876/scopej/ads -X POST -d '{"output_level":"debug"}'

# Or via istioctl
istioctl admin log --level ads:debug

For pilot-agent, the same --log_output_level flag works. Container logs land on the standard streams.

Common scopes you'll want at debug for an investigation:

Scope	What it tells you
ads	Per-stream xDS push events, NACK reasons, dependency calculations
model	Push context construction, sidecar scope resolution
serviceregistry	Service / Endpoint events, registry merges
kube	Kubernetes informer activity, watch reconnects
ca	CSR signing
injector	Sidecar injection decisions
validation	Webhook validation failures
default	The catch-all; useful when you're not sure where the issue is

Structured fields

log.WithLabels("proxy", proxyID, "type", typeURL).Infof("pushed %d resources", len(resources))

Always attach proxy and type labels in xDS code paths — the existing log lines do, and aggregations rely on them.

ControlZ

ControlZ is the in-process admin HTTP UI istiod and pilot-agent expose. Defaults:

• istiod: :9876 (configurable via --ctrlz_port)
• pilot-agent: :15022

Endpoints:

Path	Purpose
/scopej	List and tweak log scopes
/scopej/<name>	GET / POST a single scope
/metricsj	Snapshot of registered metrics
/envj	Process environment
/argsj	Process arguments
/versionj	Build info
/memj	Heap stats
/proc/pprof/...	pprof handlers

The HTML pages exist alongside the *j JSON endpoints and are convenient when port-forwarding.

istiod debug endpoints

Beyond ControlZ, istiod exposes a richer debug HTTP server (see pilot/pkg/xds/debug.go, ~1200 lines). Default port :8080. Highlights:

Path	Returns
/debug/configz	The full ConfigStore as JSON
/debug/syncz	Per-proxy sync status (last NACK, last push time)
/debug/registryz	Service and endpoint registry contents
/debug/endpointz	Computed endpoints per service
/debug/edsz	EDS snapshots being served
/debug/cdsz, /debug/ldsz, /debug/rdsz, /debug/ndsz	Computed xDS snapshots
/debug/pushcontextz	The current push context
/debug/authorizationz	Authorization policies attached to a workload
/debug/list	List of available debug endpoints
/debug/instancesz	Per-proxy Proxy state
/debug/cachez	xDS cache contents/stats
/debug/sidecarz	Effective sidecar scopes
/debug/inject	Computed injection template for a pod

If you are debugging "Envoy got the wrong config", /debug/configdumpz?proxyID=... returns the same Envoy admin config_dump that the proxy itself would, but as computed by istiod. That's how you compare what istiod thinks vs what Envoy actually has.

istioctl tooling

istioctl is the tool you reach for when investigating live data-plane issues.

Command	What it shows
istioctl proxy-status	Per-proxy ADS sync status across all xDS types
istioctl proxy-config clusters <pod>	CDS as Envoy sees it
istioctl proxy-config listeners <pod>	LDS as Envoy sees it
istioctl proxy-config routes <pod>	RDS
istioctl proxy-config endpoints <pod>	EDS
istioctl proxy-config secret <pod>	SDS / certs
istioctl proxy-config bootstrap <pod>	Bootstrap config
istioctl proxy-config log <pod> --level debug	Tweak Envoy admin log levels remotely
istioctl experimental describe pod <pod>	High-level summary of why a pod is configured the way it is
istioctl x precheck	Pre-upgrade config sanity checks
istioctl analyze	Static analysis of mesh config; finds 50+ classes of misconfiguration
istioctl bug-report	Collects diagnostics: pods, configs, logs, Envoy/ztunnel dumps
istioctl ztunnel-config <subcmd>	Ambient-mode equivalents for ztunnel pods
istioctl waypoint <subcmd>	Generate, list, or check waypoint deployments

Add -o yaml (or -o json) for machine-readable output.

Bug reports

istioctl bug-report is the official "give me everything" diagnostic. It collects:

• Pod state and resource usage across the mesh
• Istiod logs and ControlZ snapshots
• Envoy and ztunnel config dumps and logs
• Istio CRs in scope
• Mesh config and the istio ConfigMap

Output is a tarball you attach to the bug report. Implementation lives in tools/bug-report/.

In-test debugging

When debugging a failing unit test:

# Verbose output and a single test
T='-run TestName -v -count=1' make test PKG=./pilot/pkg/networking/core/...

For golden-file tests, check the diff before regenerating:

go test -run TestName ./...                     # see actual vs golden
REFRESH_GOLDEN=true go test -run TestName ./... # then regenerate

The pilot/pkg/xds/adstest.go helpers spin up a fake xDS server with deterministic config. They are the right starting point when you need to reproduce an xDS push interactively.

For controllers built on krt, the framework has built-in debugging hooks (/debug/krt on the controller's debug port, when registered). The Collection.Debug() method dumps current state.

Common control-plane symptoms

Symptom	First place to look
Envoy not getting config	istioctl proxy-status; /debug/syncz; check NACKs
Wrong endpoints	/debug/edsz?proxyID=...; kubectl get endpointslices
AuthZ denying valid traffic	istioctl x authz check; /debug/authorizationz
Sidecar not injected	istioctl x check-inject pod/<pod>; injector logs
CRD validation fails	validation log scope; admission webhook logs
Slow push or high CPU	/debug/cachez, /metrics (look for pilot_xds_push_time_seconds); turn on ads:debug
Stale resource	pilot/pkg/model/push_context.go debouncing; PILOT_DEBOUNCE_AFTER

Common data-plane symptoms

Symptom	Place to look
Connection refused 503	Envoy access log on the server sidecar; check response_flags (UH=no upstream healthy, UF=fail, NR=no route)
503 UC	Upstream connection terminated; usually app-side timeouts or mismatched protocol detection
503 UH	No healthy hosts; check EDS via istioctl proxy-config endpoints
TLS handshake failures	istioctl proxy-config secret to verify SDS; ca log scope on istiod
Ambient request hangs	istioctl ztunnel-config workload to verify ztunnel sees the workload; ztunnel pod logs

When in doubt, the istio-proxy access log is the single most-informative artifact — turn it on globally with meshConfig.accessLogFile: /dev/stdout while debugging.