Background and design notes

The repository ships with extensive in-tree design documentation under docs/architecture/. This page is a roadmap to those documents, plus a few additional notes on cross-cutting decisions.

Hand-maintained design docs

Each of these is authoritative for its topic; the wiki summarises and links rather than duplicates.

Document	Topic
docs/architecture/README.md	High-level system overview.
docs/architecture/history-service.md	History service deep dive (24KB).
docs/architecture/matching-service.md	Matching service overview.
docs/architecture/chasm.md	CHASM framework reference.
docs/architecture/nexus.md	Nexus (~21KB).
docs/architecture/workflow-lifecycle.md	Workflow lifecycle sequence diagrams.
docs/architecture/workflow-update.md	Update RPC end-to-end (~31KB).
docs/architecture/speculative-workflow-task.md	How updates and queries deliver tasks without writing history.
docs/architecture/message-protocol.md	Update message wire format.
docs/architecture/schedules.md	Scheduling (~20KB).
docs/architecture/retry.md	Activity retry policies.
docs/architecture/effect-package.md	Pre-CHASM "effect" mechanism.
docs/architecture/in-memory-queue.md	The in-memory scheduled queue used by some queue families.
docs/architecture/worker-commands.md	Worker command dispatch in History.
docs/architecture/worker-versioning.md	Worker versioning v1/v2 model (file lives at the docs/ root).
service/matching/fairness.md	Matching fairness scheduler design.

When the wiki and an in-tree doc disagree, the in-tree doc is authoritative.

Pitfalls / danger zones

A non-exhaustive list of places where the engineering team has been bitten more than once:

• Shard handoff and RangeID — every shard write must carry the current RangeID. Forgetting this in a new code path is the most common cause of "two hosts both think they own the shard" bugs.
• Mutable State invariants — every state change that emits an event also emits / cancels a corresponding history task. The matching is enforced manually; missing it means the engine wedges.
• N-DC merge edge cases — the conflict resolver in service/history/ndc/ handles version-history divergence; new event types must be tested against multi-cluster scenarios.
• Queue ack levels — periodic checkpointing trades startup-cost for write-amplification; tuning the period without testing leads to slow restarts or excessive writes.
• Standby executors — every active executor needs a standby variant that performs the same verification but doesn't drive the workflow forward. A missing standby manifests as replication lag.

Migration context

A few "this used to work differently" notes that surface in code review:

• HSM → CHASM — both coexist (see HSM, CHASM). New state machines should use CHASM unless a maintainer has a reason to use HSM.
• Worker versioning v1 → v2 → v3 — the public API still references all three. New work should target v3.
• Schedules — the classic implementation in service/worker/scheduler/ is being superseded by chasm/lib/scheduler/.
• Visibility on Elasticsearch — many older clusters used the primary DB for visibility; ES is now the recommended path for production.

Cross-cutting decisions

• Why fixed shard count? — re-sharding is expensive and rarely needed if the cluster is sized at creation. Operators add capacity by adding hosts, not shards.
• Why event sourcing + Mutable State summary? — pure event-sourcing replay would be too slow for most workflow decisions; the summary is the cache.
• Why Cassandra was the original backend? — write-heavy, eventually-consistent persistence aligned with the event-sourcing model.
• Why fx? — typed dependency injection scales better than init() functions in a 1.7k-file codebase; the fx graph is also testable.
• Why no rewrite to a different language? — the SDK ecosystem and the existing operator base are deeply Go-friendly. The team has stuck with Go and continues to invest in framework abstractions (CHASM) over language migrations.