Logging

Where logs go

By default the server writes structured logs to stdout. The log: block of the YAML config tunes:

log:
  stdout: true
  level: info
  outputFile: ''
  format: json

The implementation is common/log/, Uber zap underneath. In production deployments operators typically forward stdout to a log aggregator (Loki, ELK, Cloud Logging).

Log levels and conventions

Level	Used for
Fatal	Invariant violations the process cannot recover from. Crashes the server.
DPanic	"This should never happen, but the cluster shouldn't crash because of it." Panics in dev / test only.
Error	Errors that another part of the system is going to retry. Single events.
Warn	Unusual but recoverable conditions; deprecation notices.
Info	State changes that operators want to see (service start, shard acquired, namespace cache refreshed).
Debug	Per-request detail. Off by default.

AGENTS.md codifies the Fatal / DPanic distinction and the Patterns and conventions → error handling page mirrors it.

Structured fields

All logs are tagged with structured fields from common/log/tag/tags.go. Common ones:

• tag.WorkflowID / tag.RunID / tag.WorkflowNamespace
• tag.ShardID / tag.ShardRangeID
• tag.TaskQueueName / tag.TaskQueueType
• tag.RequestID
• tag.Error

These names are stable across releases — log queries written against them don't break with refactors. When adding a new tag, add it to tags.go; don't pass loose zap.String calls into hot paths.

Sampling

Hot paths use log.ThrottledLogger (common/log/throttle_logger.go) to drop repeated messages above a threshold. Sampling rates are configured per category via dynamic config (system.sampledLogging*).

Common log patterns

Message	What to do
"Range conditional update failed."	Benign during shard handoff; persistent occurrence means flap.
"Membership update received."	Normal during deploys.
"Closing shard."	Shard releasing ownership; expected when host leaves ring.
"Persistence Error" with Persistence:Cassandra	Inspect the wrapped DB error; usually a network or driver issue.
"Workflow execution already started"	Caller's reuse policy collided with an existing workflow.

Adding a log statement

logger.Info("acquired shard",
    tag.ShardID(s.id),
    tag.ShardRangeID(s.rangeID),
)

• Use a tag rather than a printf-style message when you can.
• Don't log payloads.
• Sample if you'll log this on a hot path.