Metrics

Where metrics are defined

Every metric in the server is declared in common/metrics/metric_defs.go. This is deliberate — central declaration keeps dashboards stable and avoids name collisions.

A typical declaration:

// in common/metrics/metric_defs.go
var ServiceRequestsScope = NewCounterDef(
    "service_requests",
    WithDescription("RPCs received per service+API"),
)

A consumer:

metricsHandler.Counter(metrics.ServiceRequestsScope.Name()).Record(1)

Backends

common/metrics abstracts the backend behind a Handler interface. Supported:

• Prometheus (pull-based, exposed on the per-service metrics endpoint).
• OpenTelemetry (OTLP push).
• Statsd (go-statsd-client).
• No-op (for tests).

The backend is selected by the global.metrics block in YAML.

Standard tags

Most metrics carry a small, low-cardinality set of tags:

• service_name — frontend, history, matching, worker.
• operation — the API method or internal task type.
• namespace — the workflow's namespace (when applicable).

High-cardinality tags (workflow ID, run ID, shard ID) are deliberately avoided; cardinality of millions per series breaks Prometheus.

Key SLIs to watch

Metric (substring)	What it tells you
service_requests	Per-RPC count.
service_latency	Per-RPC latency histogram.
service_errors_*	Error counts split by error class.
persistence_latency	Persistence backend latency.
persistence_errors_*	Errors from persistence; baseline-watch this.
task_latency_load / task_latency_processing	History queue task processing time.
task_redispatched	How often a queue task is redelivered.
replication_dlq_max_message_id	Replication DLQ depth.
shard_controller_*	Shard acquisition/release events.
task_queue_added / task_queue_polled	Matching service throughput.

Adding a metric

1. Add a *Def in common/metrics/metric_defs.go with a clear description.
2. Use the typed MetricsHandler (Counter, Histogram, Gauge) in service code.
3. Don't roll a Prometheus client manually — go through the central handler.

Where to read more

• Logging, Tracing
• Systems → Telemetry
• Reference → configuration