Debugging

How to figure out what's happening when something is wrong, broken down by component.

Logging

Every component uses k8s.io/klog/v2 (re-exported and configured by staging/src/k8s.io/component-base/logs). Verbosity is controlled by --v=N:

Level	Roughly what you see
0	Errors only
1	Plus warnings and high-signal info (default)
2	Plus per-loop status and reconciliation summaries
3	Plus per-call API request and response summaries
4	Plus per-object diffs and decision-making detail
5	Plus controller-internal trace events
6	Plus all HTTP request/response headers
7+	Full body dumps; use sparingly

JSON logging is opt-in via --logging-format=json. The format is registered at boot by importing _ "k8s.io/component-base/logs/json/register".

Common errors and where to start

Symptom	First thing to check
kubectl apply returns 404 for a resource	The right apiVersion is enabled in the API server. Look at --runtime-config and the pkg/registry/<group>/ wiring.
kubectl apply returns forbidden	Authentication / RBAC. Run kubectl auth can-i .... Check the audit log on the apiserver.
Pod stuck in Pending	The scheduler has rejected it. kubectl describe pod shows the per-plugin reasons. Bump scheduler verbosity or look at pkg/scheduler/schedule_one.go flow.
Pod stuck in ContainerCreating	The kubelet is failing somewhere between "image pull" and "sandbox create". Look at kubelet logs, then runtime (crictl ps -a, crictl pods). Mount failures point to volume-plugin issues.
Pod runs but Service VIP is unreachable	kube-proxy data plane. Inspect iptables-save, nft list ruleset, or ipvsadm on the node. Check the proxy mode in --proxy-mode.
Controller not reacting to changes	Informer not getting events. Verify watch is open (kubectl get --watch), confirm RBAC for the controller's service account, look for informer.HasSynced deadlocks.
API request is slow	Check apiserver /metrics for apiserver_request_duration_seconds, look at etcd latency, check for noisy clients with apiserver_flowcontrol_* metrics.

Inspecting a running cluster

kubectl get --raw /readyz?verbose          # apiserver subsystem readiness
kubectl get --raw /livez?verbose
kubectl get --raw /metrics                 # apiserver Prometheus metrics
kubectl get --raw /api/v1/nodes/<node>/proxy/metrics  # kubelet metrics (in-cluster auth)
kubectl get --raw /api/v1/nodes/<node>/proxy/stats    # kubelet stats
crictl ps -a; crictl pods                   # CRI-level, on a node

Dumping internal state

The controller-manager and scheduler expose /debug/pprof if launched with --profiling=true (default):

kubectl get --raw /api/v1/namespaces/kube-system/pods/<scheduler-pod>/proxy/debug/pprof/heap

The kubelet has the same endpoints under :10250/debug/pprof/.

Reproducing in a debugger

For unit tests:

dlv test ./pkg/scheduler/... -- -test.run=TestSchedulePodFails

For a running component, attach to the process or build with make GOLDFLAGS=" " GOGCFLAGS="all=-N -l" to retain symbols.

Reading the source

When the symptom isn't enough, the next move is usually:

1. Find the metric or log line. git grep "thing-the-log-said" is fast on this repo.
2. Walk up to the goroutine entry point. Most loops are named Run, worker, syncFoo, or processNextItem.
3. Find the informer that feeds the loop. It's almost always created in the same file as Run.
4. Find the reconcile function. Naming convention: syncHandler, reconcile, processItem. The body usually shows the read-from-cache, compute-diff, write-to-API pattern.

Reporting bugs upstream

1. Search the issue tracker for an existing report.
2. Reproduce with the smallest possible YAML and the latest minor release. Many issues are fixed in newer patches.
3. Include component logs at --v=4, kubelet crictl ps -a, and the relevant CRDs / RBAC.
4. Use the kind/bug label and the SIG label that owns the affected area.

For a structured incident-style debugging walkthrough of common failures (apiserver crash, etcd full disk, watch storm), see the SIG-Cluster-Lifecycle troubleshooting guide linked from the README.