pulumi/pulumi

Backends

Active contributors: Thomas Gummerer, Justin Van Patten, Fraser Waters, Ian Wahbe

Purpose

A backend is where Pulumi stores its state — the snapshot of resources, the stack metadata, the encrypted secrets, the deployment history. The repo ships two implementations:

DIY (pkg/backend/diy/) — a "do it yourself" backend that reads/writes plain files. Supports filesystem, S3, GCS, Azure Blob, and any gocloud.dev/blob URL.
Pulumi Cloud / httpstate (pkg/backend/httpstate/) — the hosted backend at app.pulumi.com. Adds API tokens, deployment history, role-based access, and ESC integration.

Both implementations satisfy the Backend interface in pkg/backend/backend.go.

Directory layout

pkg/backend/
├── backend.go                 # Backend interface (≈21 KB) — the contract
├── stack.go                   # Stack interface
├── snapshot.go                # Snapshot persistence (≈39 KB)
├── apply.go                   # Apply machinery shared by ops
├── journal.go                 # Journaled snapshot persistence (≈30 KB)
├── updates.go                 # Update history
├── watch.go                   # `pulumi watch`
├── secrets.go                 # Secrets manager wiring
├── policypack.go              # Policy-pack interface
├── organizations.go           # Org list
├── cloud_registry.go          # Hint about cloud-provided registries
├── mock.go                    # In-memory mock (used in tests)
├── cancellation_scope.go      # Cancellation propagation
├── validating_persister.go    # Snapshot integrity guard
├── backenderr/                # Domain-specific error types
├── display/                   # Event renderer (TUI + JSON)
├── secrets/                   # Backend-aware secrets-manager glue
├── state/                     # State file utilities
├── diy/                       # DIY backend (filesystem / blob)
└── httpstate/                 # Pulumi Cloud backend

The `Backend` interface (abridged)

From pkg/backend/backend.go:

type Backend interface {
    Name() string
    URL() string
    CurrentUser() (string, []string, *workspace.TokenInformation, error)
    GetPolicyPack(...) (PolicyPack, error)

    ParseStackReference(s string) (StackReference, error)
    GetStack(...) (Stack, error)
    CreateStack(...) (Stack, error)
    RemoveStack(...) (bool, error)
    RenameStack(...) (StackReference, error)
    ListStacks(...) ([]StackSummary, ContinuationToken, error)

    Apply(...) (*ApplyResult, error)
    Watch(...) error
    GetLatestConfiguration(...) (config.Map, error)
    GetHistory(...) ([]UpdateInfo, error)

    DefaultSecretManager(...) (secrets.Manager, error)
    // ...
}

The CLI never reaches into a backend's storage layer directly. It calls these methods, the backend handles details.

DIY backend (`pkg/backend/diy/`)

graph LR
    CLI[pulumi CLI] -->|backend.Apply| DIY[diy.Backend]
    DIY -->|read/write| Bucket[gocloud.dev/blob bucket]
    Bucket -->|file| FS[Local FS<br/>file://&#126;/.pulumi]
    Bucket -->|s3| S3[Amazon S3]
    Bucket -->|gs| GCS[Google Cloud Storage]
    Bucket -->|azblob| AZ[Azure Blob Storage]

Files live under <bucket>/.pulumi/:

.pulumi/
├── meta.yaml                  # version + format markers
├── stacks/<project>/<stack>.json   # current snapshot
├── history/<project>/<stack>/      # update history
├── locks/                          # cooperative locking
└── ...

The DIY backend uses cooperative locking (pkg/backend/diy/lock.go) — multiple pulumi up invocations against the same stack will refuse to step on each other.

Key files:

File	Purpose
`pkg/backend/diy/backend.go` (~52 KB)	The `Backend` impl
`pkg/backend/diy/bucket.go`	Blob bucket wrapper
`pkg/backend/diy/lock.go`	Inter-process locking
`pkg/backend/diy/state.go` (~21 KB)	Snapshot read/write
`pkg/backend/diy/store.go` (~12 KB)	Stack store
`pkg/backend/diy/stack_tags.go`	Tag persistence (newly added)
`pkg/backend/diy/postgres/`	Experimental Postgres backend

backend_legacy_test.go exists to keep older on-disk formats readable.

Pulumi Cloud backend (`pkg/backend/httpstate/`)

The hosted backend talks to app.pulumi.com via REST + a small streaming layer for live engine events.

File	Purpose
`pkg/backend/httpstate/backend.go` (~93 KB)	Main backend impl
`pkg/backend/httpstate/client/`	Generated/handwritten REST client
`pkg/backend/httpstate/state.go` (~14 KB)	Snapshot upload/download
`pkg/backend/httpstate/diffs.go`	Diff computation for cloud rendering
`pkg/backend/httpstate/policypack.go` (~19 KB)	Policy-pack publish/list
`pkg/backend/httpstate/snapshot.go`	Cloud snapshot reads
`pkg/backend/httpstate/snapshot_benchmark_test.go`	Performance regression test
`pkg/backend/httpstate/token_source.go`	OAuth/token refresh
`pkg/backend/httpstate/cloud_registry.go`	Cloud-managed registry hints
`pkg/backend/httpstate/journal/`	Cloud-side journaling for resume
`pkg/backend/httpstate/environments.go`	ESC integration

Pulumi Cloud also supports managed deployments — running pulumi up on Pulumi's infrastructure rather than the user's. The CLI pulumi deployment subcommand (pkg/cmd/pulumi/deployment/) is the surface; the cloud-side execution happens server-side and emits events back through the same channel.

Backend selection

Selection happens in pkg/cmd/pulumi/backend/. Logic in order:

--backend-url flag overrides everything.
Persisted credentials (~/.pulumi/credentials.json) determine the default.
pulumi login writes credentials; pulumi login --local uses DIY at ~/.pulumi.
pulumi login s3://bucket (etc.) selects DIY with a remote bucket.

Snapshot integrity

The validating_persister.go wraps the chosen backend so every snapshot write is round-tripped through Snapshot.VerifyIntegrity() before being persisted. A snapshot with dangling parent URNs, missing providers, or duplicate URNs is rejected — preventing a buggy operation from poisoning future runs.

Entry points for modification

Adding a new backend (rare) — implement Backend, Stack, StackReference. Mirror DIY for the simplest case.
Adding a new file in DIY — touch pkg/backend/diy/store.go for path conventions.
Adding a new Cloud REST endpoint — pkg/backend/httpstate/client/. Don't forget to update the integration tests in tests/login/ if the endpoint affects login flow.
Changing snapshot format — extremely high risk. Any change must be backwards-compatible with pkg/backend/diy/backend_legacy_test.go baselines and the metaschema in sdk/go/common/apitype/.