Telemetry

Active contributors: david, alex, samar

Purpose

Three orthogonal observability surfaces, all wired up from common/:

• Logs — common/log/, Uber zap with structured fields.
• Metrics — common/metrics/, centrally-defined names with Prometheus / OpenTelemetry / statsd backends.
• Traces — common/telemetry/, OpenTelemetry SDK with OTLP exporters.

The three are independent but conventionally correlated by request-id, namespace-id, workflow-id, and run-id tags emitted everywhere.

Logging

Loggers come from fx. The base logger is built in common/log/zap_logger.go. Conventions:

• Tags only via tag.* — the helpers in common/log/tag/tags.go define stable field names. Don't pass loose zap.String calls into hot paths.
• Sampling — common/log/ThrottledLogger drops repeated messages above a threshold.
• Severity — see Patterns and conventions → error handling.

Metrics

common/metrics defines:

• A central registry of metric names in metric_defs.go. Adding a new metric means editing this file (and adding documentation strings; they are scraped to build internal dashboards).
• A Handler interface that abstracts the backend (Prometheus, statsd, OTLP, no-op).
• Common interceptor helpers (grpc.go, tally_metrics_handler.go) so service code rarely emits metrics directly — most observability is automatic per-RPC.

Metric tags are kept low-cardinality. Per-namespace and per-RPC tags are common; per-workflow and per-shard tags are not.

Traces

OpenTelemetry is wired through common/telemetry/ and the otel.* blocks in YAML config. Notable features:

• gRPC server interceptors via go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.
• Persistence operations are wrapped in spans by common/persistence/telemetry/.
• Custom span names follow temporal.<service>.<operation>.

Configuration

Logging and metrics are configured under the top-level log: and global.metrics: blocks of the YAML config (see config/development-sqlite.yaml). Tracing is configured under global.tracerProvider.

Health checks

Each service exposes the standard gRPC health-check service (common/rpc/interceptor/health.go) plus a deeper internal check that walks shard ownership / persistence / membership. The History service has the most elaborate one in service/history/deep_health_check.go.

Related pages

• How to monitor — operator-focused guide.
• How to contribute → debugging
• Reference → configuration