Tracing

etcd has two layers of tracing:

1. traceutil — a lightweight, in-process trace recorder used by the apply pipeline and the v3 server.
2. OpenTelemetry — optional distributed tracing that exports spans to an OTLP collector.

traceutil

Source: pkg/traceutil/trace.go.

Every request that enters EtcdServer.Range / Put / DeleteRange / Txn carries a *traceutil.Trace in its context. As the request walks through the apply pipeline, callers add trace.Step("step name") markers with timestamps. If the total request exceeds traceThreshold (100 ms in server/etcdserver/v3_server.go), the trace is logged with all the steps:

{"level":"warn","ts":"...","caller":"v3_server.go:...","msg":"apply request took too long",
 "took":"125.4ms","expected-duration":"100ms","prefix":"read-only range",
 "request":"key:\"k\" ", "response":"range_response_count:1 size:24",
 "trace":[{"step":"agreement","took":"75ms"},{"step":"applied index","took":"2ms"}, ...]}

This is the primary forensic tool when "etcd is slow"; no external infrastructure required.

OpenTelemetry

Source: server/embed/config_tracing.go, server/etcdserver/tracing.go.

Activated via:

--experimental-enable-distributed-tracing=true
--experimental-distributed-tracing-address=otel-collector:4317
--experimental-distributed-tracing-service-name=etcd
--experimental-distributed-tracing-sampling-rate-per-million=10000

Defaults (server/embed/config.go):

• DefaultDistributedTracingAddress = "localhost:4317"
• DefaultDistributedTracingServiceName = "etcd"

The exporter uses OTLP/gRPC over go.opentelemetry.io/otel/exporters/otlp/otlptrace. Spans are emitted at:

• gRPC server interceptors (one span per request).
• apply/ boundaries (one span per applied entry).
• bbolt commit (in server/storage/backend/).

How spans connect

graph LR
    grpc[gRPC interceptor span] --> server[EtcdServer span]
    server --> apply[apply span]
    apply --> backend[backend.Commit span]
    apply --> mvcc[mvcc.Put span]

The gRPC traceparent metadata header propagates from clients into the server, so etcd participates in your existing distributed-trace context.

Operator tips

• Run a low sampling rate in production (e.g. 10/M = 0.001%) and bump it during incidents.
• Combine with the Prometheus metrics: a slow span correlates with etcd_server_apply_duration_seconds.
• Trace data does not include keys/values (PII safety).

Cross-references

• how-to-contribute/debugging — turning on debug logs and pprof.
• systems/apply — the applier emits the most-watched spans.