Persistence

Active contributors: alex, david, samar

Purpose

common/persistence is the pluggable storage layer behind every Temporal service. Every workflow's state, every task queue's pending tasks, every namespace and shard record passes through these interfaces. It is the single largest sub-tree under common/ (407 Go files) and is the hardest part of the codebase to change without breaking compatibility.

Directory layout

common/persistence/
├── data_interfaces.go              # The Go-level persistence contract (50KB)
├── data_interfaces_mock.go         # Generated mock for unit tests
├── execution_manager.go            # Per-shard workflow execution operations
├── history_manager.go              # History event read/write
├── metadata_manager.go             # Namespace metadata
├── namespace_replication_queue.go
├── nexus_endpoint_manager.go
├── shard_manager.go
├── task_manager.go                 # User-facing matching tasks
├── persistence_interface.go        # Internal/lower-level interface (35KB)
├── persistence_metric_clients.go   # Wrapper that emits metrics
├── persistence_rate_limited_clients.go  # Wrapper that rate-limits per-namespace
├── persistence_retryable_clients.go     # Wrapper that retries idempotent ops
├── operation_mode_validator.go     # Enforces the mode rules (Create/Update/etc.)
├── size.go / size_util.go          # Per-call payload size accounting
│
├── cassandra/                      # Cassandra implementation
├── sql/                            # SQL implementation (MySQL, PG, SQLite)
│   ├── execution.go
│   └── sqlplugin/
│       ├── mysql/
│       ├── postgresql/
│       └── sqlite/
├── nosql/                          # No-op generic NoSQL adapter (currently Cassandra-only)
├── faultinjection/                 # Wrapper that injects errors for testing
├── visibility/                     # Visibility store (separate, with ES variant)
├── client/                         # Factory + client construction
├── serialization/                  # Proto serialisation helpers
├── persistence-tests/              # Cross-backend persistence test suite
├── persistencetest/
├── schema/                         # Schema-change utilities
├── transitionhistory/
├── versionhistory/
├── telemetry/                      # Tracing wrappers
├── tests/
└── mock/

Key abstractions

Type / file	What it is
ExecutionManager (common/persistence/execution_manager.go)	Per-shard workflow execution + task operations.
HistoryManager (common/persistence/history_manager.go)	History event tree read/write.
MetadataManager (common/persistence/metadata_manager.go)	Namespace metadata.
TaskManager (common/persistence/task_manager.go)	Matching task queue tasks.
ShardManager (common/persistence/shard_manager.go)	Shard ownership rows + RangeID writes.
DataStoreFactory (common/persistence/client/)	Builds backend-specific implementations.
VisibilityManager (common/persistence/visibility/)	Separate manager for the visibility store (ES, SQL, or in-DB).

Wrapper stack

Every operation goes through a chain of wrappers in this order (constructed in common/persistence/client/ and applied in temporal/fx.go):

graph LR
    Caller --> Telemetry[telemetry tracer]
    Telemetry --> Metrics[persistence_metric_clients]
    Metrics --> RateLimit[persistence_rate_limited_clients]
    RateLimit --> Retry[persistence_retryable_clients]
    Retry --> FaultInj[faultinjection<br/>tests only]
    FaultInj --> Backend[Cassandra / SQL / NoSQL]

The order is deliberate: rate-limit BEFORE retry, so retries don't hammer through the limiter; metrics OUTSIDE retry, so a single logical operation reports one metric event regardless of how many retries it took.

Backends

Cassandra (common/persistence/cassandra/)

The original backend. Schema in schema/cassandra/. Many of the schema choices (row-per-execution, per-shard locking via UPDATE … IF) were forced by Cassandra's lack of multi-row transactions and propagated to the other backends for consistency.

SQL (common/persistence/sql/)

A common SQL implementation parameterised on a "plugin" interface that handles dialect differences. Plugins:

• sqlplugin/mysql/
• sqlplugin/postgresql/
• sqlplugin/sqlite/

SQLite is the developer-friendliest backend and ships embedded — make start-sqlite-file runs the entire cluster against a local file. The schema for each dialect is under schema/{mysql,postgresql,sqlite}/.

Visibility (common/persistence/visibility/)

A separate manager because the visibility store is logically distinct (often a different database — Elasticsearch). Implementations:

• Standard SQL visibility (uses the same SQL plugin set as the primary store).
• Elasticsearch via go.olivere/elastic/v7.
• A "dual" visibility manager that writes to both during migrations.

Schema versioning

Schema changes go through temporal-cassandra-tool (cmd/tools/cassandra/) or temporal-sql-tool (cmd/tools/sql/). New columns or tables get a new version directory under schema/<backend>/<dbname>/versioned/. The setup target embeds a schema_version table that the tool reads to determine which migrations to apply.

Operation modes

Some operations are conditional: "create only if not exists" or "update only if version matches". The operation_mode_validator.go enforces correctness by verifying that the caller's mode matches the persistence row's state. A mode mismatch triggers a *persistence.ConditionFailedError, which is the engine's primary mechanism for optimistic-concurrency retry.

Testing

• Unit tests use the generated mock from data_interfaces_mock.go.
• Persistence integration tests under persistence-tests/ run the same test suite against every backend.
• Fault injection wraps real backends with random errors via faultinjection/; used to exercise the retry path.
• Schema migration tests are in schema/ and run via the schema tools.

Entry points for modification

• Adding a query — add to data_interfaces.go, regenerate mocks, then implement for each backend (Cassandra + SQL plugins). The persistence integration test suite runs the new query against every backend.
• Adding a column — write a new migration in schema/<backend>/<dbname>/versioned/ and update the corresponding ORM-style mapping in cassandra/ or sql/.
• Adding a backend — implement DataStoreFactory in common/persistence/client/ and the various manager interfaces. Then add a SQL plugin or NoSQL adapter alongside the existing ones.

Related pages

• History service — the dominant persistence consumer.
• Matching service — owns task queue persistence.
• Replication — uses the namespace-replication queue.
• Reference → data models — a brief catalogue of the major tables.