Debugging

Plugins built for debugging

CoreDNS ships three debugging-oriented plugins. Loading them in a Corefile gives you better visibility:

• debug (plugin/debug) — turns on log.Debug* output and disables panic recovery. With debug loaded, a plugin panic crashes the process so you get a usable stack trace; without it, panics are swallowed, logged, and replied to with SERVFAIL.
• pprof (plugin/pprof) — exposes /debug/pprof on a configurable port. Use this for CPU and heap profiling.
• log (plugin/log) — per-query access logs.

Pair errors with log and you get one log line per request and one log line per plugin error.

A debugging Corefile:

.:1053 {
    debug
    log
    errors
    pprof :6060
    forward . 8.8.8.8
}

Logging conventions

plugin/pkg/log produces lines of the form:

[INFO] plugin/forward: 1 upstreams marked unhealthy
[ERROR] plugin/forward: connection error: dial udp 1.2.3.4: i/o timeout
[DEBUG] plugin/cache: prefetch a.example.com.
[WARNING] plugin/loop: forwarding loop detected for "."

The [DEBUG] lines only appear when the debug plugin is loaded.

A common mistake: using the standard library log package instead of plugin/pkg/log. The latter prefixes the plugin name and respects the global debug toggle. CI lint will not catch this, but reviewers will.

Common failure modes

Listen on :53 requires elevated privileges

listen udp :53: bind: permission denied

Pick a non-privileged port (:1053) for development:

.:1053 { ... }

Or grant the binary the right capabilities:

sudo setcap CAP_NET_BIND_SERVICE=+eip ./coredns

"no next plugin found"

[ERROR] plugin/forward: no next plugin found

A plugin called plugin.NextOrFailure but its Next is nil. Usually means a backend plugin couldn't answer and there's nothing after it in the chain. Add a fallback (forward . 8.8.8.8) at the end of the server block.

"cannot serve - it is already defined"

Two unfiltered server blocks claim the same transport://address:port. Either consolidate the blocks or add a view filter to one of them.

"Failed to start: listen tcp 0.0.0.0:9153: bind: address already in use"

The Prometheus listener (default :9153) is being claimed by something else. Move it (prometheus :9100) or kill the conflicting process. Note that the metrics plugin is process-wide; loading it in two server blocks doesn't double-bind, but loading it on different ports does.

Slow first query, fast subsequent queries

forward opens persistent connections lazily. The first query pays the dial latency. Use health_check to keep upstreams warm, or set a higher expire to reuse connections longer.

kubernetes plugin returns SERVFAIL

Often a startup-sync issue. The plugin reports not-ready until the initial Service/Endpoint informer caches populate. Load the ready plugin and check /ready:

curl -i http://localhost:8181/ready

If /ready is 503 long after startup, the API server is reachable but the cache populate is failing — check kubectl get events, RBAC for the service account, and the apiserver_qps setting.

Plugin order surprises

The Corefile order is irrelevant. The active order is in zdirectives.go. If you load both cache and forward and the cache isn't behaving, check that you didn't accidentally rely on the wrong plugin order. Run coredns -plugins to see the compiled-in list.

pprof profiling

. {
    pprof :6060
    ...
}

Then:

go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
go tool pprof http://localhost:6060/debug/pprof/heap
go tool pprof -alloc_space http://localhost:6060/debug/pprof/allocs

Goroutine dump:

curl -s http://localhost:6060/debug/pprof/goroutine?debug=2

dnstap for query/response capture

When you need to capture upstream exchanges:

. {
    dnstap unix:///tmp/dnstap.sock full
    forward . 8.8.8.8
}

full includes payloads. Read the socket with the official dnstap CLI tool to inspect each forward and the upstream's response.

Race detector

go test -race ./plugin/cache/...

A race in cache or forward is the most common surprise after a refactor. Plugins manipulate shared maps and connection pools; lock acquisition order matters.

Trace mode

The trace plugin (Observability) plus a Zipkin or Datadog backend gives you per-plugin spans for each query. Useful for figuring out which plugin in a long chain is the slow one.

Common gotchas

• Mutating without copying. A plugin that edits a dns.Msg in place corrupts the upstream cache or zone data. Always Copy() first. See Patterns and conventions.
• Forgetting make gen. Adding a plugin to plugin.cfg without regenerating zplugin.go and zdirectives.go produces silently broken builds. The verify-make-gen workflow catches it in CI.
• Logging at INFO from hot paths. log.Info is unconditional; on a busy server it floods stdout. Use log.Debug and require the debug plugin to enable it.
• reuseport only matters on Linux. macOS/BSD users hitting "address in use" during reload should expect this — multisocket and graceful socket handoff are Linux-specific.