Observability middlewares

These middlewares are not user-configurable per router — they are inserted by the framework around every router chain. The pkg/server/middleware/observability.go builder decides what gets wrapped and in what order. The runtime hooks are in pkg/middlewares/.

For the broader picture (logging, metrics, and tracing as systems), see Observability.

Recovery

pkg/middlewares/recovery/. Always at the top. Catches panic from any downstream handler and converts it into a 500 response with an error log line. Without it, a single panicking middleware would take down the goroutine handling the connection.

Capture

pkg/middlewares/capture/. Wraps the response writer with a counter for bytes, status, and time. Stashes a Capture value in the request context so any later middleware or the access-log middleware can read latency/bytes without re-instrumenting.

Capture exposes:

• StatusCode() — final status, including 200 if not explicitly written.
• RequestBytes(), ResponseBytes() — counts.
• Latency() — accumulated time after the handler completes.

Tracing

pkg/middlewares/observability/tracing.go (and friends in the same directory). Starts a span for each incoming request, propagates W3C traceparent headers to upstream requests, and records middleware events as span attributes when a middleware exposes:

type tracingInformer interface {
    GetTracingInformation() (name string, typeName string, kind trace.SpanKind)
}

Most user middlewares implement GetTracingInformation() so the span tree shows the chain. The tracer itself is constructed by pkg/observability/tracing/tracing.go and handed to the wrapper at startup.

pkg/middlewares/observability/ also contains the metrics-tied wrappers and the OTel propagator selection (TraceContext, Baggage, B3, Jaeger).

Metrics

pkg/middlewares/metrics/. Reads the captured per-request data and updates the configured metrics registry. The metrics emitted depend on the registry type but always include:

• Per entry point: traefik_entrypoint_requests_total{code,method,protocol}.
• Per router: traefik_router_requests_total{code,method,protocol,router}.
• Per service: traefik_service_requests_total{code,method,protocol,service} and a duration histogram.

The middleware is added once per layer (entry point, router, service) — pkg/server/middleware/observability.go decides where each one goes.

Access log

pkg/middlewares/accesslog/. Produces one log line per request, in either CLF or JSON format. The work is split:

• accesslog.go — main entry point; records request and response data.
• log_handler.go — async writer goroutine with a buffered channel.
• parser.go, clf.go, json_formatter.go — output format encoders.
• field_apply.go — selects which fields to include based on configuration.

Captured fields: time, level, method, host, path, status, content length, latency, client IP (post PROXY/X-Forwarded-For), user agent, request ID, router name, service name, retry count, downstream status, and configured custom headers.

Handler switcher

pkg/middlewares/handler_switcher.go. Strictly speaking not a middleware — it's a wrapper around http.Handler that holds an *atomic.Value. Configuration apply swaps the value; concurrent requests see the new handler tree without any locking.

Response modifier

pkg/middlewares/response_modifier.go. A small helper used by middlewares that need to alter the response after it has been written (e.g. adjusting headers added by upstream). It implements the http.ResponseWriter interception trick and exposes hooks the wrapping middleware uses.

Stateful

pkg/middlewares/stateful.go. A tiny utility wrapper that lets middlewares declare they need to capture additional state per request (e.g. for response modification). It's referenced from a handful of middlewares and not user-facing.

Putting it together

graph TD
    EP[Entry point] --> Recovery
    Recovery --> Capture
    Capture --> EpMetrics[entry-point metrics]
    EpMetrics --> Tracing[tracing wrapper]
    Tracing --> RouterMetrics[router metrics]
    RouterMetrics --> User[user middleware chain]
    User --> SvcMetrics[service metrics]
    SvcMetrics --> AccessLog[access log]
    AccessLog --> Service[service handler]

The exact insertion order is enforced by pkg/server/middleware/observability.go and is not configurable. This is what makes observability "free" — every router gets the same instrumentation by construction.