Observability

Logging, metrics, and tracing in Traefik are three connected systems that share one principle: every observability concern is implemented as a middleware that automatically wraps user-defined chains.

Three flavors

Concern	Static config	Implementation packages
Logs (server + access)	Log, AccessLog	pkg/observability/logs, pkg/middlewares/accesslog
Metrics	Metrics	pkg/observability/metrics, pkg/middlewares/metrics
Tracing	Tracing	pkg/observability/tracing, pkg/middlewares/observability

Server logs

pkg/observability/logs/logs.go configures the global zerolog logger. It is the only place that mutates log.Logger. All other code uses contextual loggers via log.Ctx(ctx).

Two log streams are emitted:

• Traefik logs — internal server logs (configuration apply, provider errors, lifecycle).
• Access logs — one line per HTTP request, by the accesslog middleware.

The Log static-config block controls level (DEBUG, INFO, WARN, ERROR), format (common or json), and output destination (stdout, stderr, or a file path with optional rotation via lumberjack-style settings).

The AccessLog block controls similar parameters for the access log stream.

Access log middleware

pkg/middlewares/accesslog is a stateful middleware. It captures:

• Request method, path, host, query, protocol.
• Response status, content length.
• Latency (time to first byte and total).
• Source IP (after PROXY-protocol and X-Forwarded-* resolution).
• Selected request and response headers, configurable per field.
• The router and service names that handled the request.

Output formats:

• CLF (Common Log Format): Apache-style host - user [time] "method url proto" status bytes.
• JSON: structured, friendly to log shippers.

The middleware writes asynchronously through a buffered channel; pkg/middlewares/accesslog/log_handler.go runs the writer goroutine.

Capture middleware

pkg/middlewares/capture is a special middleware injected by pkg/server/middleware/observability.go. It records bytes in/out and elapsed time and stashes them in the request context so downstream middlewares (access log, metrics, tracing) can read them without re-instrumenting.

Metrics

pkg/observability/metrics/metrics.go defines a typed metrics interface:

type Registry interface {
    IsEpEnabled() bool
    IsRouterEnabled() bool
    IsServiceEnabled() bool
    EntryPointReqsCounter() metrics.Counter
    EntryPointReqsTLSCounter() metrics.Counter
    EntryPointReqDurationHistogram() metrics.Histogram
    EntryPointOpenConnsGauge() metrics.Gauge
    RouterReqsCounter() metrics.Counter
    // ...
    ServiceReqsCounter() metrics.Counter
    ServiceReqDurationHistogram() metrics.Histogram
    ServiceServerUpGauge() metrics.Gauge
    // ...
}

The interface lets multiple registries coexist. Backends:

File	Backend
pkg/observability/metrics/prometheus.go	Prometheus pull-style.
pkg/observability/metrics/otel.go	OpenTelemetry OTLP push.
pkg/observability/metrics/datadog.go	DogStatsD UDP.
pkg/observability/metrics/statsd.go	StatsD UDP.
pkg/observability/metrics/influxdb2.go	InfluxDB v2 line protocol.

Each backend has its own Stop*() function, all called by stopMetricsClients() in pkg/server/server.go during shutdown.

The pkg/middlewares/metrics middleware reads the captured per-request data and increments counters / observes histograms. The metrics middleware lives in the chain produced by pkg/server/middleware/observability.go.

What's measured

• Per entry point: requests, requests TLS, request duration, open connections, sent/received bytes.
• Per router: requests, request duration, open connections.
• Per service: requests, request duration, open connections, retries, server up/down (one gauge per upstream).
• TLS specifics: certificate not after, certificate not after by domain.
• Configuration reload counters and timestamps.

The full surface is the Registry interface; metrics_test.go enumerates expected counters per backend.

Tracing

pkg/observability/tracing/tracing.go builds an OpenTelemetry tracer:

func NewTracer(ctx, conf *otypes.Tracing) (trace.Tracer, io.Closer, error)

Configuration options accepted:

• OTLP exporter via gRPC or HTTP.
• Sampling rate.
• Resource attributes (service name, environment).
• Optional capture of headers, query, and body — be careful with PII.

The tracer is then handed to pkg/middlewares/observability/tracing.go, which:

• Starts a span for each incoming request, sets the standard HTTP server attributes.
• Propagates the W3C traceparent header on the upstream request.
• Records middleware-level events as span events when the middleware exposes GetTracingInformation().

Tracing in middlewares and the proxy

Each middleware that wants to add a span calls tracing.StartSpan(req.Context(), m.name, ...). The proxy (pkg/proxy/fast or pkg/proxy/httputil) instruments the upstream request automatically via the round-tripper.

Health and ping

Two endpoints round out the observability surface:

• GET /ping — liveness, served by pkg/ping. Returns OK while the process is healthy.
• GET /metrics — metrics, served by Prometheus when enabled.

The traefik healthcheck subcommand (cmd/healthcheck) calls /ping so Docker's HEALTHCHECK can do the same with the Traefik binary itself.

Wiring summary

graph TD
    StaticConf[static.Configuration] --> Logs
    StaticConf --> Metrics
    StaticConf --> Tracing

    Logs[pkg/observability/logs] --> Logger[zerolog logger]
    Metrics --> Registry[*metrics.MultiRegistry]
    Tracing --> Tracer[OTel Tracer]

    Logger --> Mids[middleware chain]
    Registry --> Mids
    Tracer --> Mids

    Mids -->|Capture middleware| Request[Request handling]
    Mids -->|AccessLog| AccessOut[(access log file)]
    Mids -->|Metrics| Backend["Prometheus / OTLP / Datadog / StatsD / InfluxDB"]
    Mids -->|Tracing| OTLP[(OTLP collector)]

Entry points for modification

• Add a new metrics backend: implement Registry in pkg/observability/metrics/<backend>.go, register it in pkg/observability/metrics/metrics.go, and add a Stop<Backend>() function called from stopMetricsClients().
• Add a new field to access logs: extend the access-log entry struct in pkg/middlewares/accesslog/, update the encoder for both CLF and JSON.
• Add tracing for a new middleware: implement GetTracingInformation() returning the middleware name and type. The observability wrapper does the rest.