Tracing

In addition to logs and metrics, every Rust service can export OpenTelemetry traces. This is opt-in via env vars.

Enabling traces

Set OTEL_EXPORTER_OTLP_ENDPOINT (and optionally OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME) when starting a service:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
OTEL_SERVICE_NAME=pageserver-prod-1 \
cargo run -p pageserver

The setup uses the opentelemetry-otlp crate with the http-proto feature (so spans go over HTTP/Protobuf, not gRPC). Implementation: libs/tracing-utils/src/lib.rs.

What gets traced

The same tracing::span! instrumentation that produces structured logs also produces OTel spans when the OTLP exporter is active. Specifically:

• HTTP requests (tower-otel integration on axum routes) → root spans.
• libpq replication-stream connections → manually created root spans.
• Background tasks (compaction, WAL ingest, eviction) → root spans of the form compaction{tenant_id=..., level=L0}.

Inside those, child spans are created for sub-operations: layer-map lookup, remote download, WAL-redo round-trip, etc.

Cross-service propagation

W3C TraceContext propagation is supported. The proxy reads traceparent from incoming HTTP requests and propagates it into the libpq connection (via the request_id/correlation field). The pageserver reads it from libpq metadata where present and uses it to nest its spans under the proxy's parent span.

This means a single trace in your OTel UI can include spans from:

• Proxy → request handling.
• Compute → query execution (only if the compute itself is instrumented; default builds are not).
• Pageserver → GetPage@LSN resolution.
• Pageserver → remote-storage download.
• Pageserver → WAL-redo round-trip.

Sampling

By default, traces are sampled with a constant ratio. The exporter respects:

• OTEL_TRACES_SAMPLER=traceidratio
• OTEL_TRACES_SAMPLER_ARG=0.01

Pick a low ratio (≤1%) for production unless you have a high-throughput trace backend.

Disabling traces

OTEL_SDK_DISABLED=true disables OTel entirely. Some integration tests set this to keep the test output quiet.

Pitfalls

• Span overhead. Creating a span on every call to a hot inner function is expensive. The pageserver leaves the hottest paths (per-page reads inside a layer scan) unspanned.
• Async + spans. Always use .instrument(span) to attach a span to a future, not just let _enter = span.enter(). Failing to do so loses the span across .await points.

Where to look in code

• libs/tracing-utils/src/lib.rs — the OTel pipeline setup.
• Each service's main.rs calls tracing_utils::init_tracing(...) at startup.
• tower-otel is added to axum middleware stacks for HTTP services.

See also

• Logging — log records also carry span context.
• Metrics
• Patterns and conventions — tracing crate usage.