Internal Worker service

Active contributors: alex, yichaoyang, david

Purpose

The Worker service runs internal system workflows — work the cluster does on behalf of itself. It uses the public Temporal SDK (go.temporal.io/sdk) just like any user worker, but registers system workflows defined inside this repository. There is exactly one Worker service per cluster (it can be horizontally scaled, but no shard-style state ownership).

Directory layout

service/worker/
├── service.go                    # Service bootstrap (16KB)
├── worker.go                     # Per-namespace worker host
├── pernamespaceworker.go         # Namespace fan-out
├── fx.go
├── addsearchattributes/          # Workflow that adds a search attribute to ES schema
├── batcher/                      # Bulk-action workflow (terminate/signal many)
├── common/                       # Shared helpers
├── deletenamespace/              # Async namespace deletion
├── dlq/                          # Replication DLQ inspector / re-enqueue
├── dummy/                        # No-op worker for templating
├── migration/                    # Schedule / replication migration helpers
├── parentclosepolicy/            # Apply parent-close policy to children
├── replicator/                   # Pulls history events from peer clusters
├── scanner/                      # Scheduled scanners (executions, history, build-ids, taskqueue)
├── scheduler/                    # The (older) Schedule workflow
└── workerdeployment/             # Worker deployment / versioning workflows

Key abstractions

Type / file	What it is
Service (service/worker/service.go)	The fx-wired host process; spins up per-namespace workers.
perNamespaceWorker (service/worker/pernamespaceworker.go)	One Temporal SDK Worker per namespace; registers all system workflows enabled for that namespace.
Scheduler (service/worker/scheduler/)	Older schedule workflow — superseded by the CHASM-based scheduler in chasm/lib/scheduler/ but still present.
Replicator (service/worker/replicator/)	Pulls replication tasks from peer clusters and applies them locally.
Scanner (service/worker/scanner/)	Scheduled background scans (orphaned executions, history cleanup, build-id reachability).
Batcher (service/worker/batcher/)	Bulk operations workflow used by temporal batch CLI commands.
DeleteNamespace (service/worker/deletenamespace/)	Async deletion workflow used by OperatorService.DeleteNamespace.
WorkerDeployment (service/worker/workerdeployment/)	Workflows that drive the worker-versioning v3 deployment lifecycle.

How it works

graph LR
    Service[worker.Service] --> PerNS[perNamespaceWorker per namespace]
    PerNS --> SDK[temporal SDK Worker]
    SDK --> Replicator[replicator workflow]
    SDK --> Scanner[scanner workflows]
    SDK --> Batcher[batch workflow]
    SDK --> Scheduler[scheduler workflow]
    SDK --> Deletion[delete-namespace workflow]
    SDK --> WD[worker-deployment workflows]
    SDK -->|task polls| Frontend

Each system workflow is a normal Temporal workflow that registers on a system task queue (per workflow type). When the Worker host starts:

1. The fx graph in service/worker/fx.go collects all enabled workflow registrations.
2. pernamespaceworker.go creates one SDK Worker per namespace that runs the system workflows that are eligible (e.g. archival workflows only run if archival is enabled for that namespace).
3. The workflows poll the cluster's own Frontend service for tasks. They are clients of the cluster they live in.

This bootstrap is recursive — the Worker service is a Temporal client of the cluster it is part of. It works because the SDK and the Frontend both already exist in-process, but the wiring carefully avoids creating cycles in the fx graph (see how the SDK client is constructed lazily in service/worker/service.go).

Notable system workflows

Workflow	Purpose
add-search-attributes-workflow (addsearchattributes/)	Coordinates schema migration for new search attributes.
temporal-sys-batch-workflow (batcher/)	Bulk terminate / signal / cancel large workflow sets.
temporal-sys-delete-namespace-workflow (deletenamespace/)	Drains and deletes a namespace including its workflows.
temporal-sys-replication-workflow (replicator/)	Pulls history from peer clusters.
temporal-sys-scanner-workflow (scanner/)	Periodic scans of executions, history nodes, build IDs, task queues.
temporal-sys-parent-close-policy-workflow (parentclosepolicy/)	Applies parent-close policy to large fan-outs.
temporal-sys-dlq-merge-workflow (dlq/)	Re-enqueues messages from the replication DLQ.
worker-deployment-* (workerdeployment/)	Implements the worker-versioning v3 deployment APIs.
migration-* (migration/)	Cluster migration / schedule migration helpers.

Integration points

• Inbound: none directly. The Worker is a client of the cluster's Frontend.
• Outbound: Frontend (StartWorkflow / Signal / Query for system workflows; Activity calls to internal admin APIs), Persistence (some scanners read directly via common/persistence/), External clusters (replicator pulls history).

Entry points for modification

• Adding a system workflow: create a new package under service/worker/<your-workflow>/, register it in fx.go, and add it to the appropriate per-namespace registration logic in pernamespaceworker.go.
• Tuning a scanner: scanner/ holds one sub-package per scanner type. Each has its own dynamic-config gates.
• DLQ ergonomics: dlq/ implements the merge / list / replay workflows used by tdbg dlq commands.

Related pages

• Frontend — the Worker service is a Frontend client.
• Replication / XDC — the replicator workflow lives here.
• Scheduling — the older scheduler is here; the new one is in chasm/lib/scheduler/.
• Worker versioning