Observability

Tracing and metrics instrumentation. containerd exposes Prometheus metrics from a built-in plugin and OpenTelemetry tracing through a configurable processor pipeline.

Tracing

Code	Role
pkg/tracing/	Wrapper around the OTel SDK; provides tracing.StartSpan / tracing.SpanFromContext
cmd/containerd/builtins/tracing.go	Blank import that registers the tracing internal plugin
plugins/server/internal/	The internal tracing service: configures the OTLP exporter and a sampler
docs/tracing.md	User-facing setup guide

The configuration looks like:

[plugins."io.containerd.tracing.processor.v1.otlp"]
  endpoint = "http://otel-collector:4317"
  protocol = "grpc"

[plugins."io.containerd.internal.v1.tracing"]
  service_name = "containerd"
  sampling_ratio = 0.1

Spans are created at gRPC handler entry (via the otelgrpc.NewServerHandler interceptor in cmd/containerd/server/server.go) and propagated through ctx into all downstream calls. Plugins that want their own spans use tracing.StartSpan(ctx, "spanName") and defer span.End().

Metrics

Code	Role
plugins/server/metrics/	Registers the ServerPlugin that exposes Prometheus on a configured TCP address
core/metrics/	Helper types for per-namespace/per-task metrics
core/metrics/cgroups/, cgroupsv2/	cgroup statistics emitted as Prometheus counters/gauges
core/runtime/v2/	Task metrics provider via the runtime/shim ttrpc API

The metrics plugin binds to [metrics] address = "127.0.0.1:1338" (no default — must be set). It exposes:

• gRPC server metrics (request count, latency, error rate) via grpc-ecosystem/go-grpc-middleware/providers/prometheus.
• Per-task cgroup metrics (cpu, memory, blkio).
• Per-image-pull progress counters.
• Per-plugin init duration histogram.

Logging

github.com/containerd/log (a thin logrus wrapper). Severity, format, and fields are configured under [debug]. See debugging.

OOM monitoring

core/runtime/monitor.go and internal/cri/oom_linux_test.go exercise the OOM watcher. Each shim runs an OOM observer that surfaces TaskOOM events; the metrics plugin counts them.

Entry points for modification

• Add a metric: declare a new Prometheus collector in core/metrics/ and register it in the metrics plugin's Init.
• Add a span: use pkg/tracing from any plugin code path. Don't import go.opentelemetry.io/otel directly.
• Different exporter: implement TracingProcessorPlugin with the new exporter and register it instead of the OTLP one.