Archival

Active contributors: david, samar, alex

Purpose

Archival lets a cluster offload completed workflow histories and visibility records to long-term cold storage. Operators can archive to S3, GCS, or a filesystem; histories remain queryable through the same APIs after archival.

Where the source lives

common/archiver/                       # Provider abstraction + concrete archivers
├── provider/                          # Pluggable factory
├── filestore/                         # Local filesystem archiver
├── s3store/                           # AWS S3 archiver
├── gcloud/                            # GCS archiver
└── ...
service/history/archival/              # In-shard archival queue logic
service/history/archival_queue_*       # Active/standby archival task executors
service/worker/scanner/                # Periodic verification scanners

Key abstractions

Type	What it is
archiver.HistoryArchiver / archiver.VisibilityArchiver (common/archiver/)	The interface a backend must implement (Archive, Get).
provider.ArchiverProvider (common/archiver/provider/)	Factory that the History service consults to obtain the right archiver per namespace.
Archival queue executor (service/history/archival_queue_task_executor.go)	Runs as part of the History shard, dispatches archive calls.

How it works

graph TD
    WFCompleted[Workflow completes] --> ArchivalTask[Archival history task created]
    ArchivalTask -->|queue processor| Executor[archival_queue_task_executor]
    Executor -->|provider lookup| Provider[ArchiverProvider]
    Provider --> S3[S3 archiver]
    Provider --> GCS[GCS archiver]
    Provider --> FS[Filesystem archiver]

1. When a workflow reaches a terminal state, History creates an Archival task as part of the same atomic write.
2. The archival queue processor in each shard executes the task: it serialises history events into chunks and pushes them to the configured archiver.
3. Reads (GetWorkflowExecutionHistory) check both online persistence and the archive, transparently to the caller.

Visibility archival

Visibility records have their own archiver implementation that mirrors the history archiver but writes the indexed visibility row into a structured location (a JSONL file in S3 / GCS by default). The ListWorkflowExecutions paths can be configured to also query the archive.

Operator concerns

• Archival is opt-in per namespace via ArchivalConfig; operators can also force-disable globally via dynamic config.
• The scanner workflows verify that completed workflows actually got archived — useful when chasing data-loss reports.

Entry points for modification

• Adding an archiver backend: implement HistoryArchiver and VisibilityArchiver in a new sub-package under common/archiver/ and register the factory with provider/.
• Changing the archival schema: the JSON schema is defined alongside each backend; bumping it requires a migration story for older archives.

Related pages

• History service — the archival queue lives in each shard.
• Internal Worker — archival verification scanners.
• Persistence — the canonical storage that archives complement.