Debugging

Daemon logs

containerd uses github.com/containerd/log (a thin wrapper over logrus). The standard verbosity flag is --log-level (trace, debug, info, warn, error, fatal).

Typical patterns:

sudo containerd --log-level debug
journalctl -u containerd -f          # if running under systemd

The format defaults to text; switch to JSON via the daemon config:

[debug]
  level = "debug"
  format = "json"

Inside Go code, retrieve a context-scoped logger with log.G(ctx).WithField(...). The ctx carries namespace, request ID, and OpenTelemetry span information; using log.G propagates these onto the log record.

The debug socket

The plugins/server/debug plugin exposes a control socket (/run/containerd/debug.sock by default). It serves Go's net/http/pprof endpoints and a few introspection helpers:

sudo curl --unix-socket /run/containerd/debug.sock \
  http://debug/debug/pprof/goroutine?debug=2

Profile types: goroutine, heap, block, mutex, threadcreate, cmdline, profile (CPU, accepts ?seconds=N).

Introspection API

The introspection gRPC service (defined in api/services/introspection/v1/) lets clients enumerate loaded plugins and their statuses:

ctr plugins ls

Use this to confirm a built-in plugin loaded successfully (look for OK vs ERROR/SKIPPED). Plugin failures surface here even when their own logs are quiet.

Tracing

OpenTelemetry tracing can be enabled in config.toml:

[plugins."io.containerd.tracing.processor.v1.otlp"]
  endpoint = "http://localhost:4317"

[plugins."io.containerd.internal.v1.tracing"]
  service_name = "containerd"
  sampling_ratio = 1.0

The tracing builtin is registered from cmd/containerd/builtins/tracing.go. See docs/tracing.md for a worked example.

Common failure modes

Symptom	Likely cause	Where to look
failed to load required plugin <id> at startup	The plugin's dependency failed; the dependency's failure log usually appears just above	plugins/<area>/...
failed to dial /run/containerd/containerd.sock	Daemon not running or wrong --address	cmd/containerd/server/server.go
Container stuck "Created" but never starts	Snapshotter mount failed or shim crashed silently	shim log under --state directory; core/runtime/v2/manager.go
Image pull hangs or 401s	Registry auth or hosts.toml mis-config	docs/hosts.md, core/remotes/docker
GC keeps deleting an image right after pull	No lease around the operation	pkg/leases, core/leases
CRI sandbox stays "NotReady"	CNI plugin missing or cni_conf_syncer failed	internal/cri/server/cni_conf_syncer.go

Inspecting state on disk

• /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db — the bbolt metadata DB. Use bbolt CLI (in vendor/) or bolt-cli to dump it.
• /var/lib/containerd/io.containerd.snapshotter.v1.<name>/ — snapshotter data.
• /var/lib/containerd/io.containerd.content.v1.content/ — the content store layout: ingest/<id> for in-progress writes and blobs/sha256/<digest> for committed blobs.
• /run/containerd/runc/<namespace>/<id>/ — per-task runc state.

Reading shim logs

The shim logs to its own file inside the state directory unless you pass --debug to the daemon (which forwards --debug to shims). The shim log path looks like <state>/<runtime>/<namespace>/<id>/log.

Step-debugging

The Makefile supports a debug build:

make GODEBUG=1

This adds -N -l to go build so dlv can step through. Attach with:

sudo dlv attach $(pgrep containerd) ./bin/containerd