Debugging

Common debugging tools for dockerd and the test suites.

Daemon logs

dockerd logs to stderr by default; under systemd they go to journald. Set verbosity with:

dockerd --log-level=debug
# or in /etc/docker/daemon.json
{ "log-level": "debug" }

The log facade is github.com/containerd/log. Every backend method receives a context.Context, and log.G(ctx) returns a logger with request-scoped fields. Routers add module=api, method, and request-url fields automatically (see the handler in daemon/server/server.go).

Stack dumps

The daemon installs a SIGUSR1 handler on Unix that dumps all goroutines to the log. See daemon/debugtrap_unix.go and daemon/debugtrap_windows.go.

kill -SIGUSR1 $(pidof dockerd)

pprof

The daemon exposes net/http/pprof on its debug endpoint when the --debug flag (or "debug": true in the config) is set. Routes are registered by daemon/server/router/debug/. Typical use:

curl --unix-socket /var/run/docker.sock http://./debug/pprof/goroutine?debug=2

Tracing

OpenTelemetry HTTP tracing is enabled in daemon/server/server.go via otelhttp.NewHandler. Spans are emitted for each Engine API call. The CI sets up an OTel collector via otelcol-ci-config.yml for failing tests.

Common failure patterns

Daemon won't start

• Existing containerd not reachable: containerd path/address is configured under [plugins."io.containerd.grpc.v1.cri"]-style options. Check containerd-address in the daemon config.
• Socket permission: dockerd needs to listen on /var/run/docker.sock. The docker group owns it; group membership matters.
• Storage driver init: graphdriver init logs come from daemon/graphdriver/. On overlay2, mount errors typically indicate kernel mismatches or xfs without d_type=true.

Container start hangs

• Likely a libnetwork issue. Check daemon/libnetwork/sandbox.go and the bridge or overlay driver logs.
• The docker-proxy userland helper may fail if the host port is in use; the bridge driver logs port allocation.

Pull/push errors

• Live in daemon/internal/distribution/ (legacy store) or the containerd resolver path under daemon/containerd/. Look for pull_v2.go / push_v2.go log entries.

Builder errors

• Classic builder: logs prefixed by the parser stage; entry points under daemon/builder/dockerfile/.
• BuildKit: log fields include vertex and name. The integration is in daemon/internal/builder-next/.

Running a debugger

make BIND_DIR=. shell mounts the source tree into the dev container. Inside, dlv attach to the running daemon, or use DELVE_PORT from the Makefile to expose the Delve port to the host.

Logs from tests

Failed integration tests dump the daemon log into bundles/test-integration/. The journal is preserved per-suite. TESTDEBUG=1 make test-integration retains the bundles even on success.