Replication / XDC

Active contributors: wxing1292, samar, alex

Purpose

Temporal supports multi-cluster (sometimes called "XDC" — cross-data-centre) replication, in which a namespace is owned by one cluster (the "active") and replicated to one or more "standby" clusters that can take over via failover. Replication preserves history events, mutable state, and CHASM transitions across clusters.

Where the source lives

service/history/replication/        # In-shard producer + consumer machinery
service/history/ndc/                 # N-data-centre conflict resolution
service/worker/replicator/           # System workflow that pulls events from peers
common/cluster/                      # Cluster metadata cache
common/persistence/namespace_replication_queue.go
proto/internal/temporal/server/api/replication/  # Replication protos

How it works

graph LR
    A[Active cluster<br/>History] -->|replication tasks| AStream[Replication stream]
    AStream -->|gRPC| B[Standby cluster<br/>Worker→replicator workflow]
    B -->|apply| BHistory[Standby cluster<br/>History]
    BHistory -->|verify, no-op| BStandby[Standby executors]

1. Generation. When the active cluster commits a state transition, it writes a Replication Task into the shard's replication queue alongside the history events and Mutable State change. Source: service/history/replication/.
2. Streaming. A long-lived gRPC stream between History services on different clusters pulls these tasks. Implementation in replication/stream_* files.
3. Application. On the receiving side the temporal-sys-replication-workflow (a system workflow under the Internal Worker service) consumes the stream and applies events through the History service's standby code path.
4. Verification. On the standby cluster, the standby task executors (*_standby_task_executor.go in service/history/) verify that the replicated state matches what the active produced — but do not advance the workflow on their own.

N-DC conflict resolution

When a namespace fails over while events are still in flight, two clusters may have diverging history. The N-DC subsystem (service/history/ndc/) implements conflict resolution: it walks the version-history trees on both sides, finds the lowest common ancestor, and merges by reapplying the active cluster's events.

VersionHistory (common/persistence/versionhistory/) records the lineage of every event; this is what conflict resolution operates on.

DLQ

Replication tasks that cannot be applied (schema mismatch, namespace deleted, malformed event) end up in a per-shard DLQ. They are inspectable via tdbg dlq (tools/tdbg/) and re-enqueued by the service/worker/dlq/ workflows.

Cluster metadata

The cluster_metadata table (see common/cluster/ and common/persistence/cluster_metadata_store.go) holds the list of known clusters, their addresses, and which is active for each namespace. Every service caches it via common/cluster/metadata.go.

Failover is performed by an admin RPC that updates the namespace's active cluster name; the History service in the new active cluster then takes over the workflow's execution by switching its standby executors to active.

Integration points

• History. Producer of replication tasks, consumer of inbound replication.
• Internal Worker. Hosts the temporal-sys-replication-workflow and the DLQ merge workflows.
• Persistence. Replication queue lives in shard-level tables.
• Membership. Replication streams are pinned to specific shards; ownership transitions invalidate the stream.

Entry points for modification

• Adding a new replicated entity: add the entity to the per-shard replication task type registry in service/history/tasks/. Implement active and standby executors. Update the protos in proto/internal/temporal/server/api/replication/ and run make proto.
• Improving conflict resolution: N-DC code in service/history/ndc/; add cases to the merge logic in ndc_workflow.go.
• Changing the DLQ schema: the queue lives in common/persistence/history_task_queue_manager.go; replicate any changes through the tdbg CLI.

Related pages

• History service — the engine that emits and consumes replication tasks.
• Internal Worker — host of the replicator and DLQ merge workflows.
• Namespaces and multi-cluster — operator-facing model.
• Persistence → version history