Observability

Active contributors: jameshartig, miekg, superq, greenpau, Tantalor93, johnbelamaric, zouyee

Purpose

These plugins make CoreDNS observable: structured query logs, error logs, Prometheus metrics, distributed tracing, dnstap, profiling, and HTTP health/readiness endpoints. None of them affect the answer; they just describe what happened.

Plugins in this group

Plugin	Source	One-liner
log	plugin/log/	Per-query access log
errors	plugin/errors/	Pretty-print errors returned by other plugins, with rate-limited consolidation
prometheus (a.k.a. metrics)	plugin/metrics/	Expose /metrics endpoint and core counters
dnstap	plugin/dnstap/	Emit dnstap-framed events (binary structured logs)
trace	plugin/trace/	Distributed tracing (Datadog, Zipkin, OpenTelemetry)
debug	plugin/debug/	Enable log.Debug output and disable panic recovery
pprof	plugin/pprof/	Expose /debug/pprof HTTP endpoint
health	plugin/health/	/health endpoint with optional lameduck
ready	plugin/ready/	/ready endpoint that aggregates Readiness checks

How they layer

graph TD
    Q[query] --> Trace[trace span]
    Trace --> Log[log: log incoming]
    Log --> Errors[errors: handle downstream errors]
    Errors --> DnsTap[dnstap: client query event]
    DnsTap --> Chain[downstream plugins]
    Chain --> Reply[reply]
    Reply --> Metrics[prometheus: counters/histograms]
    Reply --> DnsTap2[dnstap: client response event]
    Reply --> Log2[log: log response]

Plugins in detail

log

Logs each request to stdout with a configurable format string. Variables come from plugin/pkg/replacer: {remote}, {name}, {type}, {class}, {rcode}, {rsize}, {duration}, {>id}, {>do}, {>port}, {plugin} and friends. Multiple log directives can apply to different name patterns or response classes.

Class filters: success, denial, error, all. Useful to log only NXDOMAINs in production.

errors

Wraps downstream ServeDNS calls so panics and errors get a single log line each. Rate-limit groups: identical errors that recur within a window are coalesced (consolidate <window> <pattern>). Without errors, plugin errors only show up via the recovery path in Server.ServeDNS.

prometheus

The metrics plugin runs an HTTP server on :9153 (default) exposing /metrics. Behavior:

• Initialised by plugin/metrics/setup.go. Implements caddy.Restart so reload events shut down the listener.
• Tracks: coredns_dns_requests_total, coredns_dns_responses_total, coredns_dns_request_duration_seconds, coredns_dns_request_size_bytes, coredns_dns_response_size_bytes, panics, plugin attribution, build info. The full list is in plugin/metrics/vars/report.go.
• Per-zone labels are populated as zones are added/removed from the server (AddZone, RemoveZone).
• Plugins register additional collectors with m.MustRegister(...). Each plugin's metrics file (e.g. plugin/cache/metrics.go, plugin/forward/metrics.go) is the convention.
• Supports HTTPS with the prometheus exporter-toolkit web config.
• Can run on multiple Server Blocks; the listener is process-wide so they share state.

dnstap

Emits binary dnstap frames over a Unix socket or TCP/TLS endpoint. Each plugin can opt in; the dnstap plugin holds a list of taps (plugin/dnstap/dnstap.go). The forward and kubernetes plugins call SetTapPlugin to attach themselves so upstream events are reported.

trace

OpenTracing-style tracer plugged into Server.trace. Two backends are supported: Zipkin and Datadog (via dd-trace-go). When enabled, plugin.NextOrFailure starts a child span for each downstream plugin so each query produces a complete waterfall.

debug

Two effects: disables recover() in Server.ServeDNS and toggles log.D.Set() so clog.Debug* calls produce output. Used during development.

pprof

Standard net/http/pprof exposed on a configurable address. Disabled by default for security.

health

A small HTTP server that returns 200 OK at /health. Optional lameduck DURATION delays shutdown — during the lameduck window /health keeps answering 200 OK while /ready reports not-ready, so a load balancer can drain traffic before the process exits.

A self-check runs once per second; the duration histogram (coredns_health_request_duration_seconds) is a soft signal of per-process load.

ready

Aggregates Readiness implementations across loaded plugins. kubernetes, for example, only reports ready once initial informer sync is complete. Returns 200 OK at /ready only when every registered check passes; in lameduck mode it returns 503.

Cross-plugin notes

• log and errors are typically loaded together: errors ensures error metadata is set on the writer, log prints them.
• prometheus doesn't need to be in every server block — it's process-wide. Loading it in two blocks is allowed but they share state.
• trace requires a backend; without it, the plugin no-ops.
• health and ready each run their own HTTP listener and can be on the same port if you really want.
• The plugin attribution metric depends on pluginWriter (see Plugin system). metadata collects helper values that log and others substitute via replacer.

Key source files

File	Purpose
plugin/log/log.go, plugin/log/setup.go	Access log
plugin/errors/errors.go, plugin/errors/setup.go	Error handling and consolidation
plugin/metrics/metrics.go, setup.go, handler.go, vars/	Prometheus exporter
plugin/dnstap/dnstap.go, taprw/, setup.go	dnstap emitter
plugin/trace/trace.go, zipkin.go, datadog.go	Distributed tracing
plugin/debug/debug.go	Debug toggles
plugin/pprof/pprof.go	pprof endpoint
plugin/health/health.go, health/poller.go	Health endpoint and lameduck
plugin/ready/ready.go	Readiness aggregator
plugin/pkg/replacer/	Placeholder substitution used by log and errors

Related pages

• How to monitor — operator-facing summary of all metrics, logs, and traces.
• Plugin system — Readiness, pluginWriter for metric attribution.
• Transforms — metadata populates values consumed by these plugins.