Tracing

Setup

OpenTelemetry tracing is wired through common/telemetry/. The OTLP exporters in go.mod (go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc and go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc) are the production targets.

Configuration lives under global.tracerProvider in YAML:

global:
  tracerProvider:
    serviceName: temporal-frontend
    exporter: otlp
    endpoint: otel-collector:4317
    sampleRate: 0.01

What is automatically traced

• gRPC server calls — via go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.
• Persistence operations — wrapped in spans by common/persistence/telemetry/.
• HTTP gateway requests — wrapped by otelhttp.

Span names follow the pattern temporal.<service>.<operation>.

Tracing a request end to end

A typical SDK call propagates the trace context as a gRPC header. As the request flows Frontend → History → Matching, each service contributes child spans:

SDK → Frontend.StartWorkflowExecution
        Frontend.persistence.UpdateNamespace (cache miss)
        Frontend → History.StartWorkflowExecution
            History.persistence.AppendHistoryNodes
            History.persistence.UpdateWorkflowExecution
            History → Matching.AddWorkflowTask
                Matching.persistence.CreateTask

The same trace ID can be correlated against logs and metrics — the standard zap tag tag.TraceID() is emitted on requests when tracing is on.

Sampling

Tracing is expensive at high QPS. The default sampler in common/telemetry/ is parent-based with a configurable head-rate. Operators typically run:

• Production: 0.1–1% sampling.
• Pre-prod / debug: 100% on a subset of namespaces via the samplingRules block.

Custom spans

Service code uses the standard otel.Tracer(...).Start(ctx, name) API. Don't overdo it — spans are cheap but stitching together a useful trace requires consistent naming.

Where to read more

• Logging, Metrics
• Systems → Telemetry