Search attributes and visibility

Active contributors: david, samar, alex

Purpose

Search Attributes are typed indexed key/value pairs attached to workflow executions. They make ListWorkflowExecutions queries possible. The visibility store is the separate database (Elasticsearch or SQL) that holds the index.

Source map

Surface	File / package
Search attribute helpers	common/searchattribute/
Visibility manager	common/persistence/visibility/
Visibility task executor	service/history/visibility_queue_task_executor.go
Visibility schema	schema/elasticsearch/, schema/<sql>/visibility/
Add-search-attribute migration workflow	service/worker/addsearchattributes/
Public protos	temporal.api.workflow.v1.* (in [go.temporal.io/api])
Frontend handler	service/frontend/workflow_handler.go (ListWorkflowExecutions, CountWorkflowExecutions, ScanWorkflowExecutions)

How it works

graph TD
    StateChange[Workflow state change in History] --> VTask[Visibility task created]
    VTask -->|queue processor| Exec[visibility_queue_task_executor]
    Exec -->|upsert| Vis[(Visibility store:<br/>Elasticsearch / SQL)]
    Caller[ListWorkflowExecutions] --> FE[Frontend]
    FE --> Vis

Every workflow state change that affects searchable metadata (start, status change, timer, search attribute upsert) emits a Visibility task. The History shard's visibility queue executor reads the task and writes to the visibility store.

Read paths (ListWorkflowExecutions, CountWorkflowExecutions, ScanWorkflowExecutions) are routed by Frontend straight to the visibility manager; History does not participate.

Custom search attributes

Operators register custom search attributes via OperatorService.AddSearchAttributes. The handler:

1. Updates the cluster-wide search-attribute mapping (a cluster_metadata-adjacent table).
2. For Elasticsearch backends, kicks off the add-search-attributes-workflow (service/worker/addsearchattributes/) to migrate the ES index mapping.

The Mapper interface in common/searchattribute/mapper.go is a pluggable boundary that can rename custom search attribute names per namespace — useful for multi-tenant clusters.

Visibility implementations

Backend	Implementation
Elasticsearch	Uses go.olivere/elastic/v7; index template in schema/elasticsearch/.
SQL (MySQL, PG, SQLite)	Uses the same SQL plugins as the primary store, but a separate visibility schema.
Dual	A wrapper that writes to two backends during migration.

Entry points for modification

• Adding a built-in search attribute: edit common/searchattribute/sadefs/constants.go, regenerate helper accessors via gensearchattributehelpers, update the schema for each backend.
• Tuning visibility latency: dynamic-config keys system.visibilityProcessor.* and system.visibilityArchiver.*.

Related pages

• Persistence
• Frontend service
• History service