Matching service

Active contributors: yichaoyang, david, alex

Purpose

Matching owns the user-facing Task Queues. Workers long-poll a task queue; History writes Workflow / Activity tasks into a task queue; Matching is the buffer in between, with throughput-oriented partitioning, fairness scheduling, and worker versioning. The starting-point doc is docs/architecture/matching-service.md; the design notes for the fairness scheduler are at service/matching/fairness.md.

Directory layout

service/matching/
├── service.go
├── fx.go
├── handler.go                         # gRPC handler
├── matching_engine.go                 # Per-host engine (~149KB)
├── matching_engine_interfaces.go
├── task_queue_partition_manager.go    # One per (queue,partition) (~85KB)
├── physical_task_queue_manager.go     # Backing storage for a partition (~34KB)
├── matcher.go / matcher_data.go       # Matcher core
├── pri_matcher.go / pri_*.go          # Priority/fairness scheduler variant
├── fair_*.go                          # Fairness backlog implementation
├── forwarder.go / pri_forwarder.go    # Cross-partition forwarding
├── backlog_manager.go / fair_backlog_manager.go
├── task.go / task_reader.go / task_writer.go
├── reachability.go                    # Worker-versioning reachability checks
├── version_sets.go / version_rule_helpers.go
├── user_data_manager.go               # Per-task-queue user data (versioning)
├── ratelimit_manager.go               # Per-namespace / queue rate limiting
├── nexus_endpoint_client.go           # Nexus task delivery
├── poller_history.go
├── liveness.go
├── ack_manager.go
├── workers/                           # Per-task-queue ephemeral workers
├── counter/
├── configs/
└── hooks/

Key abstractions

Type / file	What it is
matchingEngineImpl (service/matching/matching_engine.go)	Per-host engine. One instance per Matching host.
taskQueuePartitionManagerImpl (service/matching/task_queue_partition_manager.go)	Owns one (namespace, task queue, partition). Holds matcher, backlog, forwarder.
physicalTaskQueueManager (service/matching/physical_task_queue_manager.go)	Concrete backing instance for a partition (read tasks from DB, persist ack levels).
Matcher (service/matching/matcher.go)	Matches polling workers against pending tasks. The pri-matcher is the fairness-aware variant.
Forwarder (service/matching/forwarder.go)	Hands tasks / pollers to the parent partition when the local partition is idle.
BacklogManager (service/matching/backlog_manager.go)	Persistence-backed FIFO backlog of pending tasks.
userDataManager (service/matching/user_data_manager.go)	Per-queue user data (versioning rules, build IDs, etc.).

How it works

graph TD
    Worker[Worker poll]
    FE[Frontend]
    Engine[matchingEngine]
    PM[partitionManager]
    Matcher[Matcher / PriMatcher]
    Backlog[(Persistence task table)]
    History[History service]

    Worker -->|long-poll| FE
    FE --> Engine
    Engine --> PM
    PM --> Matcher
    Matcher -.->|task ready?| Backlog
    History -->|AddWorkflowTask / AddActivityTask| Engine
    Engine -->|persist task| Backlog
    PM -.->|forward when idle| ParentPM[Parent partition]

A polling worker arrives at the Frontend, gets routed to the Matching host that owns the task queue, and parks on the Matcher. When History calls AddWorkflowTask on the same partition, Matching either:

1. Hands the task directly to a parked poller (sync match — the fast path).
2. Persists the task to the backlog when no poller is available, then dispatches it later when one polls.
3. Forwards the task or poller up the partition tree (see below) to find a match.

Partitions and forwarding

A user-facing Task Queue is split into N partitions (default 4) for horizontal throughput. The root partition is the parent of all children for small N; for larger N the parents form a tree. When a child partition has tasks but no pollers, it forwards tasks (and idle pollers in the reverse direction) up the tree until a match is found. Loading any partition forces the root to load too — required so forwarding always terminates.

Fairness scheduling

The classic FIFO scheduler is implemented in matcher.go; the priority-aware variant is in pri_matcher.go and is gated on the EnablePriorityFairness dynamic-config flag. Fairness state (per-partition virtual times, per-customer queues) is in matcher_data.go and persisted via fair_backlog_manager.go and the "fair task reader/writer" pair. Design notes live alongside the code in service/matching/fairness.md.

Worker versioning

Per-task-queue versioning data (compatible build-ID sets, default versions, redirect rules) lives behind userDataManager. The version-set merge logic is in version_sets_merge.go; the versioning v3 rule helpers in version_rule_helpers.go. Reachability — "is build ID X still seeing traffic?" — is computed by reachability.go.

Integration points

• Inbound: Frontend (poll, addtask), History (add task, query reachability).
• Outbound: History (record-task-started callbacks; spawn for sticky cancellation), Nexus targets (nexus_endpoint_client.go).
• Persistence: writes / reads to the task and task_queue_user_data tables.
• Membership: task-queue partition ownership is computed via Ringpop on (namespace, task-queue, partition).

Entry points for modification

• Adding behaviour to a queue: task_queue_partition_manager.go is the orchestrator. New behaviours typically slot in around pollTask / addTask.
• Tuning the matcher: matcher.go and pri_matcher.go; both have extensive tests in *_test.go. The simulator in cmd/tools/fairsim/ is the right tool for evaluating fairness changes before running them in production.
• Versioning rules: version_rule_helpers.go, version_sets.go, and the related public API in temporalio/api.
• Per-task-queue config knobs: dynamic-config keys in common/dynamicconfig/constants.go prefixed matching.*.

Related pages

• Frontend — entry point for polls and add-task RPCs.
• History — generates Transfer Tasks that become Add Task RPCs.
• Worker versioning
• Task queues primitive
• Persistence