Tracing

Purpose

tracing/tracing.go configures OpenTelemetry tracing for the Prometheus server. Spans are emitted around scrape, query, and rule-evaluation operations and forwarded to a configurable OTLP endpoint.

File

tracing/
├── tracing.go      # Setup + reload helper.
├── tracing_test.go
└── testdata/

Configuration

Configured under tracing: in prometheus.yml (also exposed via per-flag CLI):

tracing:
  endpoint: tempo.observability.svc.cluster.local:4317
  client_type: grpc # or http
  insecure: true
  timeout: 10s
  sampling_fraction: 0.1
  tls_config: { ... }
  headers: { ... }

The provider is a sdktrace.TracerProvider with a batch span processor and an OTLP gRPC or HTTP exporter.

Spans emitted

• query_range / query_instant (in web/api/v1/api.go).
• query_eval, inner_eval, subquery_eval (in promql/engine.go).
• scrape (in scrape/scrape.go) — per scrape iteration with target labels as span attributes.
• rule.exec (in rules/group.go) with rule.kind=alerting|recording.
• remote_storage_send (in storage/remote/queue_manager.go).
• notify_send (in notifier/sendloop.go).

Trace ID in logs

When tracing is enabled, the engine adds a traceID field to query log entries. The 3.11 fix #18189 ensured this works correctly — previously only spanID was emitted.

Reload semantics

tracing.Tracer().ApplyConfig rebuilds the exporter when the tracing block changes. The 3.11 bugfix #18469 corrects an OTLP HTTP startup failure when insecure: true is set.

Self-metrics

• prometheus_tracing_dropped_spans_total
• prometheus_tracing_export_duration_seconds

Entry points for modification

• New span: add otel.Tracer("github.com/prometheus/prometheus/<area>") and tracer.Start(ctx, "name") in the relevant package; remember to set the span status on error.
• New exporter: the OTLP exporter is hard-wired; supporting Jaeger or Zipkin would mean adding a new branch in tracing.go::buildExporter.

See Debugging for how operators consume the spans.