Debugging

How to find out what is actually happening in a Temporal cluster, locally and in production.

Logging

Temporal uses Uber zap wrapped by the project's own common/log/ package. Loggers are obtained via fx and tagged with structured fields from common/log/tag/.

Key facts:

• Log level is configured under log.level in the server config (see config/development-sqlite.yaml).
• Hot paths use sampled logging via log.NewSampledLogger to avoid flooding under high QPS.
• The convention from AGENTS.md is:
  • logger.Fatal — only for invariant violations the process cannot recover from.
  • logger.DPanic — for "this should never happen but should not crash production". Panics in dev / test.
  • logger.Error — for actionable, retried-elsewhere problems.
  • logger.Warn / logger.Info — for state changes and rare events.

Search for log fields by their tag (tag.WorkflowID(...), tag.ShardID(...)) in common/log/tag/tags.go. The fields are stable, so log queries written against them keep working across refactors.

Common runtime errors

Error	Where it happens	What it means
serviceerror.ShardOwnershipLost	History service	The current host no longer owns the shard. The caller should retry against the new owner.
serviceerror.WorkflowExecutionAlreadyStarted	Frontend / History	A WorkflowID collision under the namespace's reuse policy.
serviceerror.NamespaceNotFound	Frontend / History	Caller referenced a namespace that doesn't exist or is deleted.
persistence.ConditionFailedError	History / Persistence	Optimistic concurrency control fired — usually benign; the caller retries.
serviceerror.ResourceExhausted	Frontend / History / Matching	A rate-limiter rejected the request. See common/quotas/.
serviceerror.DeadlineExceeded	any	RPC timed out. Often surfaces a downstream issue rather than the immediate caller.

A complete catalogue is in common/serviceerror/. When you see an unfamiliar one, grep there first.

Diagnosing a stuck workflow

The workflow appears to be running but is making no progress. Diagnostic order:

1. Check Mutable State. Use the operator CLI:

   tdbg workflow describe -n <namespace> -w <workflow-id>

   Source: tools/tdbg/. This reads MutableState straight from persistence.

2. Check the history. temporal workflow show returns the public history; missing events usually mean the workflow is stuck waiting on a Workflow Task or activity timeout that hasn't fired.

3. Check shard tasks. tdbg shard list-tasks --shard-id <id> lists pending Transfer / Timer / Visibility tasks. A backlog usually means a queue processor is throttled or stuck.

4. Check Matching backlog. temporal task-queue describe shows whether a Workflow Task is sitting unpolled.

5. Check replication DLQ. If the workflow is in a multi-cluster setup, replication failures end up in the DLQ. Inspect with tdbg dlq or via the worker UI.

In-process debugging

For local debugging, run the server under a debugger:

dlv debug ./cmd/server -- --env development-sqlite --allow-no-auth start

Hotspots that are useful to set breakpoints in:

• service/history/handler.go — every History RPC starts here.
• service/history/api/update_workflow_util.go — the shared state-transition path.
• service/history/transfer_queue_active_task_executor.go — Transfer task execution.
• service/matching/matching_engine.go — Matching task queue logic.

pprof and tracing

The server exposes pprof on the port configured by pprof.port in YAML. Enable it in dev configs and grab CPU / heap profiles:

go tool pprof http://localhost:7936/debug/pprof/heap

OpenTelemetry tracing is wired up in common/telemetry/; export endpoints are configured under the otel block in YAML. See How to monitor → tracing.

Replication and XDC

Replication issues are the most common production-debugging surface that escape local development. The dedicated docs at docs/architecture/ (e.g. nothing dedicated to replication, but history-service.md covers the mechanics) plus the source under service/history/replication/ and service/worker/replicator/ are the references. Symptoms to recognise:

• A workflow is "stuck" in standby mode → standby task processor is behind. Check tdbg dlq list --dlq-type replication.
• Cross-cluster failover does not promote the standby → check cluster_metadata table contents and the membership rings on both sides.

When things crash

• Look at the panic stacktrace; the relevant frame is usually inside service/history/ or chasm/.
• Check for RangeID mismatch lines around the time of the crash — these indicate a shard handoff happened, which is benign.
• Persistence-layer panics during shutdown are the most common false-alarm; the executor was draining when persistence returned an unexpected error code.

See Patterns and conventions → error handling for guidance on what to log vs panic vs return.