Pod sync state machine

How kubelet keeps a single pod's actual state in sync with its desired state. This is the most subtle part of the kubelet because of three races: the API server changes the desired state, the runtime changes the actual state, and the user can kubectl delete mid-flight.

The actors

• Pod source mux (pkg/kubelet/config/) — folds three streams into one channel of PodUpdate events:
  • API server (filtered by spec.nodeName)
  • Static-pod manifest directory (/etc/kubernetes/manifests by default)
  • Static-pod HTTP source (legacy)
• Kubelet.syncLoop (pkg/kubelet/kubelet.go) — selects on PodUpdate, PLEG events, periodic timer, eviction, container GC, and dispatches to pod workers.
• Pod workers (pkg/kubelet/pod_workers.go) — one goroutine per pod UID. The worker owns the lifecycle and serializes operations on its pod.
• PLEG (pkg/kubelet/pleg/generic.go) — Pod Lifecycle Event Generator. Periodically lists containers from the runtime, computes deltas vs the previous list, and emits ContainerStarted / ContainerDied / ContainerRemoved events.
• Status manager (pkg/kubelet/status/) — the only goroutine that writes Pod.Status. It serializes status updates and Patches the API server.

State transitions

stateDiagram-v2
    [*] --> Pending: PodUpdate(create)
    Pending --> Syncing: SyncPod(create)
    Syncing --> Running: containers up, status reported
    Running --> Syncing: spec change<br/>or PLEG event<br/>or periodic resync
    Running --> Terminating: PodUpdate(delete)<br/>or eviction<br/>or completion
    Terminating --> Terminated: all containers stopped
    Terminated --> Removed: status reported,<br/>pod deleted from API
    Removed --> [*]

The actual code uses these worker states (pkg/kubelet/pod_workers.go):

• SyncPod — pod should be running. Reconcile against runtime.
• TerminatingPod — pod should be stopped but containers may still be running. Issue stop, wait for grace period.
• TerminatedPod — pod is fully stopped. Clean up resources, finalize status, allow eviction.
• PodWorkerSync vs PodWorkerTerminating vs PodWorkerTerminated — the worker's idea of where a pod is in its lifecycle, used to drop or coalesce updates that arrive in the wrong order.

What SyncPod does

Kubelet.SyncPod (in pkg/kubelet/kubelet.go) is the per-pod reconciliation function. The worker calls it for every event affecting the pod. SyncPod:

1. Generates the API API status (generateAPIPodStatus) by combining the kubelet's view, runtime status, prober results, and volume status.
2. Reports status via the status manager.
3. Runs admission checks (e.g. eviction admission, sysctl, runtime-class allowlist) — if admission fails, the pod is rejected with a reason.
4. Ensures the pod cgroup exists (Container Manager).
5. Ensures the pod data dir exists.
6. Sets up sandbox + waits for IP allocation.
7. Pulls images.
8. Mounts volumes (driven by the volume manager, but SyncPod waits for it).
9. Computes the diff between desired containers and runtime containers; calls runtime to start/kill/exec hooks.
10. Schedules probes via the prober manager.

What PLEG does

PLEG runs every second by default (relistPeriod). It:

1. Lists every PodSandbox + Container from the runtime.
2. Compares against its previous snapshot.
3. Emits events for each transition.
4. The kubelet sync loop maps each event back to a pod UID and triggers a sync.

Why PLEG and not pure event subscription? Because the CRI's container streams aren't reliable across runtime restarts. Periodic relisting is the source of truth.

A blocked PLEG (for example, if the runtime is hung) is the most common kubelet pathology. The watchdog (pkg/kubelet/watchdog/) and the pleg.HealthChecker log warnings and eventually mark the kubelet NotReady.

Status updates

Only the status manager writes Pod.Status. The flow:

1. SyncPod composes a PodStatus and calls kl.statusManager.SetPodStatus(pod, status).
2. Status manager queues a Patch.
3. The patch sender goroutine drains the queue, deduplicates, and Patches the API server.
4. On Patch success, it caches the API view to avoid spurious updates.

The two-write-path approach (SyncPod composes, status manager writes) is what allows the kubelet to keep working when the apiserver is briefly unreachable: SyncPod still updates internal state and the status manager retries with backoff.

Common pitfalls

• Stuck Terminating — an unrelated finalizer is holding the API object. The kubelet has stopped containers and would happily clean up, but the apiserver returns the object on watches because the finalizer set is non-empty.
• PLEG is not healthy — runtime is slow or hung. Look at runtime logs first; the kubelet itself is rarely the cause.
• Phantom containers — PLEG sees a container the API has already deleted. Eventually reaped by container GC (pkg/kubelet/kubelet_pods.go plus pkg/kubelet/container/runtime.go).

Key source files

File	Purpose
pkg/kubelet/pod_workers.go	Per-pod worker state machine
pkg/kubelet/kubelet.go	syncLoop, SyncPod, admission pipeline
pkg/kubelet/kubelet_pods.go	Pod-list reconciliation, hostname/aliases, cgroup parents
pkg/kubelet/pleg/generic.go	Pod lifecycle event generation
pkg/kubelet/status/status_manager.go	The only writer to Pod.Status
pkg/kubelet/lifecycle/predicate.go	Pod admission
pkg/kubelet/kuberuntime/kuberuntime_manager.go	CRI client + container kill/start orchestration