kubelet

The kubelet is the per-node agent. It registers the node with the API server, watches for pods scheduled to itself, and drives a CRI runtime to make those pods exist (or not). It is the largest single Go package in the repo: pkg/kubelet/ alone holds a six-figure number of lines of Go.

Directory layout

cmd/kubelet/
├── kubelet.go                       # main(), 5 lines
└── app/
    ├── server.go                    # NewKubeletCommand, run loop
    ├── options/                     # KubeletConfiguration flags + KubeletConfig file
    └── ...

pkg/kubelet/
├── kubelet.go                       # The Kubelet struct (>150 KLoC), entry point
├── kubelet_pods.go                  # Pod sync loop, lifecycle reconciliation
├── pod_workers.go                   # One worker per pod; runs the sync state machine
├── kubelet_node_status.go           # Node registration + heartbeat
├── kubelet_volumes.go, volume_host.go
├── kubelet_network*.go              # Pod-network plumbing (kubelet's view of CNI)
├── kubelet_resources.go             # Capacity / allocatable arithmetic
├── kubelet_server_journal.go        # journalctl pass-through
├── reason_cache.go                  # Recent failure reasons surfaced in pod status
├── runtime.go                       # The CRI/runtime interface
├── allocation/                      # Resource allocation manager (CPU manager, memory manager, topology manager)
├── apis/                            # Kubelet's own internal APIs (PodResources gRPC, etc.)
├── cadvisor/                        # cgroup + container metrics
├── certificate/                     # Bootstrap and rotate kubelet client + serving certs
├── checkpointmanager/               # On-disk state checkpoint helpers
├── client/                          # API server client
├── clustertrustbundle/              # ClusterTrustBundle projection
├── cm/                              # Container Manager (cgroups, devices, topology, OOM, hugepages)
├── config/                          # Static-pod and pod-source config muxer
├── configmap/, secret/              # Per-pod config/secret manager
├── container/                       # Generic runtime abstractions and helpers
├── envvars/                         # Service env-var generator
├── eviction/                        # Memory/disk pressure eviction
├── images/                          # Image manager: pull, gc
├── kubeletconfig/                   # KubeletConfiguration loader (file + drop-ins + dynamic)
├── kuberuntime/                     # Concrete CRI client + state machine ⇄ kubelet
├── lifecycle/                       # PodAdmit handlers, lifecycle handlers (PostStart/PreStop)
├── logs/                            # Container log providers (CRI-streaming + sidecars)
├── metrics/                         # Prometheus metrics
├── network/                         # Probe-able pod network status
├── nodeshutdown/                    # Graceful shutdown manager (systemd inhibitor)
├── nodestatus/                      # Node Conditions + Status helpers
├── oom/                             # OOM watcher
├── pleg/                            # Pod Lifecycle Event Generator (relisting drives most reconciliation)
├── pluginmanager/                   # CSI plugin and device-plugin registration
├── pod/                             # Pod manager — kubelet's view of "what pods exist here"
├── podcertificate/                  # PodCertificate projection
├── preemption/                      # Critical-pod admission preemption
├── prober/                          # Liveness, readiness, startup probes
├── qos/                             # QoS class helpers
├── runtimeclass/                    # RuntimeClass enforcement
├── server/                          # The kubelet HTTPS server (logs, exec, attach, port-forward)
├── stats/                           # Container/pod/node stats summary
├── status/                          # Pod status manager (the only writer to Pod.Status)
├── sysctl/                          # Allowed unsafe sysctls
├── token/                           # Bound service-account token cache
├── types/                           # Shared kubelet types
├── userns/                          # User namespaces (alpha)
├── util/                            # Many helpers; cgroup, mount, PID file
├── volumemanager/                   # Per-pod volume mount/unmount loop
├── watchdog/                        # Self-watchdog and PLEG health checker
└── winstats/                        # Windows perf counters

Boot sequence

1. main() in cmd/kubelet/kubelet.go calls app.NewKubeletCommand(ctx).
2. Flag + config-file parsing. The KubeletConfiguration (in pkg/kubelet/apis/config) is loaded from disk and overlaid with drop-in files and command-line flags.
3. Cert bootstrap. If the kubelet has no client cert, it submits a CSR (signed by the cluster CA) and waits for approval.
4. Construct dependencies: cAdvisor, container runtime client (CRI), volume plugins, cgroup driver, image manager, eviction manager, plugin manager (device + CSI plugin sockets), prober manager, OOM watcher.
5. Run — the main loop is in Kubelet.Run (in pkg/kubelet/kubelet.go). It starts: PLEG, status manager, prober, volume manager, eviction, plugin manager, image GC, container GC, server (HTTPS), Node status reporter.
6. Sync loop (kubelet.syncLoop) — drives pod reconciliation forever from pod-source events, PLEG events, periodic resyncs, eviction triggers.

How a pod becomes running

graph TD
    APIs[kube-apiserver] -->|watch Pods| PodConfig[pkg/kubelet/config<br/>pod source mux]
    Static[Static-pod manifest dir] --> PodConfig
    HTTP[Static-pod HTTP source] --> PodConfig
    PodConfig -->|PodUpdate| SyncLoop[Kubelet.syncLoop]
    SyncLoop -->|dispatch| Workers[pod_workers.go<br/>one goroutine per pod]
    Workers -->|SyncPod| Runtime[kuberuntime]
    Runtime -->|gRPC| CRI[Container runtime]
    CRI --> CNI[CNI plugin]
    Workers --> Volumes[volumemanager]
    Volumes --> Mount[mount/CSI/in-tree]
    Workers --> Status[status manager]
    Status -->|Patch /status| APIs
    Runtime -.relist.-> PLEG
    PLEG -->|PodLifecycleEvent| SyncLoop

The pod-workers state machine in pkg/kubelet/pod_workers.go is the heart of the kubelet. Each pod has an associated worker goroutine; the worker owns the lifecycle (sync, terminating, terminated, removed). The state-transition logic — handling races between API updates, PLEG events, and explicit terminations — is the most-debugged code in the kubelet.

What the kubelet does not do

• It doesn't talk to etcd. Only kube-apiserver does.
• It doesn't decide where pods go. The scheduler does.
• It doesn't program the Service data plane. kube-proxy does.
• It doesn't pull from the API for arbitrary objects — it strictly watches Node, Pod (filtered by spec.nodeName), CSIDriver, Lease (for heartbeats), and a handful of others.

Subsystem deep dives

• Pod sync state machine — how pod_workers and PLEG keep pods in sync
• Container Manager (cm/) — cgroups, devices, topology, OOM
• Volume management — kubelet's volume mount/unmount machinery

Key source files

File	Purpose
cmd/kubelet/app/server.go	NewKubeletCommand, dependency wiring, Run
pkg/kubelet/kubelet.go	Kubelet struct, syncLoop, PodAdmit pipeline
pkg/kubelet/kubelet_pods.go	Pod create/update/delete reconciliation
pkg/kubelet/pod_workers.go	Per-pod worker state machine
pkg/kubelet/kubelet_node_status.go	Node registration + heartbeat
pkg/kubelet/pleg/generic.go	Pod Lifecycle Event Generator
pkg/kubelet/kuberuntime/kuberuntime_manager.go	The high-level CRI client
pkg/kubelet/cm/container_manager_linux.go	Cgroup hierarchy and resource managers
pkg/kubelet/volumemanager/volume_manager.go	Volume reconciler
pkg/kubelet/server/server.go	The HTTPS server (10250)

Integration points

• CRI runtime — gRPC, defined in staging/src/k8s.io/cri-api, consumed in pkg/kubelet/kuberuntime/.
• CNI — invoked by the runtime, but kubelet observes the result via pkg/kubelet/network/.
• CSI — kubelet talks to per-node driver sockets via pkg/volume/csi/.
• Device plugins — registered through pkg/kubelet/pluginmanager/ and consumed by the device manager in pkg/kubelet/cm/devicemanager/.
• kube-proxy — independent peer, not invoked by kubelet directly.
• Eviction — triggers from cAdvisor stats, signals back into the pod-workers via pkg/kubelet/eviction/.

Entry points for modification

• New CRI feature: extend staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto, regenerate, then plumb through pkg/kubelet/kuberuntime/.
• New volume plugin: implement the pkg/volume.VolumePlugin interface and register in pkg/volume/plugins.go. Most new volume types should ship as out-of-tree CSI drivers instead.
• New eviction signal: add to pkg/kubelet/eviction/types.go, implement the threshold reader, register in the manager.
• New admission check: implement lifecycle.PodAdmitHandler and register via kubelet.AddPodAdmitHandler.