pulumi/pulumi

Deployment engine

Active contributors: Ian Wahbe, Thomas Gummerer, Fraser Ashyggton, Julien, Justin Van Patten

Purpose

The deployment engine is what turns "the user wants this state" into "Pulumi performed these actions". It plans (diffs old vs desired), executes (creates/updates/deletes resources via providers), and persists (writes a new snapshot). Every Pulumi operation — up, preview, refresh, destroy, import — goes through it.

It has two halves:

pkg/engine/ — the operation drivers and event plumbing.
pkg/resource/deploy/ — the actual deployment: source iteration, step generation, step execution, resource graph.

Directory layout

pkg/engine/
├── engine.go                  # The Engine type — top-level entry
├── update.go                  # `pulumi up` (≈44 KB)
├── destroy.go                 # `pulumi destroy`
├── refresh.go                 # `pulumi refresh`
├── import.go                  # `pulumi import`
├── deployment.go              # Configures a Deployment for an op
├── events.go                  # All engine event types (≈32 KB)
├── eventsink.go               # diag → engine event bridge
├── plugins.go                 # Plugin discovery + lifecycle (≈30 KB)
├── policypacks.go             # Policy-pack analyzers
├── progress.go                # Progress reporter for the display layer
├── plugin_host.go             # Plugin host helper
├── snapshot.go                # Snapshot loading
├── journal_snapshot.go        # Crash-safe journaled snapshot writer (≈33 KB)
├── combined_manager.go        # Multi-snapshot composition
├── debugging.go               # `--attach-debugger` plumbing
├── install_manager.go         # Plugin install orchestration
└── lifecycletest/             # Property-based fuzz test harness

pkg/resource/deploy/
├── deployment.go              # A single deployment instance (≈30 KB)
├── deployment_executor.go     # Orchestrates step gen + exec (≈38 KB)
├── source.go                  # Source iterator interface
├── source_eval.go             # Eval source: drives a real language host (≈123 KB — the big one)
├── source_null.go             # Null source: empty program, used by destroy
├── step.go                    # Step state machine (≈88 KB)
├── step_generator.go          # Diffs and emits steps (≈137 KB — biggest non-generated file)
├── step_executor.go           # Runs steps in dependency order
├── snapshot.go                # In-memory snapshot manipulation
├── analyze_snapshot.go        # Snapshot health + integrity
├── builtins.go                # Built-in invokes (e.g. stack reference)
├── plan.go                    # Plans (`pulumi up --plan`)
├── import.go                  # Resource import logic
├── resource_hooks.go          # Lifecycle hooks
├── resource_options.go        # Per-resource option flags
├── target.go                  # `--target` filter logic
├── state_builder.go           # Immutable state transitions
├── providers/                 # Default-provider materialization
└── deploytest/                # Mock provider used by lifecycle tests

How a single resource gets created

sequenceDiagram
    participant Prog as User program
    participant Mon as ResourceMonitor (in engine)
    participant SrcEval as source_eval.go
    participant StepGen as step_generator.go
    participant StepExec as step_executor.go
    participant Prov as Provider plugin
    participant Snap as Snapshot

    Prog->>Mon: RegisterResource(type, name, inputs)
    Mon->>SrcEval: emit RegisterEvent
    SrcEval->>StepGen: SourceEvent
    StepGen->>Snap: read prior state
    Snap-->>StepGen: old state (may be nil)
    StepGen->>Prov: Check(inputs)
    Prov-->>StepGen: validated inputs
    StepGen->>Prov: Diff(old, new)
    Prov-->>StepGen: diff result
    StepGen->>StepExec: emit CreateStep / UpdateStep / SameStep
    StepExec->>Prov: Create / Update / no-op
    Prov-->>StepExec: id, outputs
    StepExec->>Snap: write new state
    StepExec-->>Mon: completion
    Mon-->>Prog: outputs

The interesting subtlety: step_generator.go is single-threaded (it owns the dependency ordering), while step_executor.go runs steps in parallel up to a configurable bound, respecting cross-resource dependencies.

Key abstractions

Symbol	File	Role
`Engine`	`pkg/engine/engine.go`	Top-level entry
`UpdateInfo`, `QueryInfo`	`pkg/engine/update.go`	Inputs to operations
`Update`, `Destroy`, `Refresh`, `Preview`, `Import`	`pkg/engine/{update,destroy,refresh,import}.go`	Operation drivers
`Deployment`	`pkg/resource/deploy/deployment.go`	One running deployment
`Source` interface	`pkg/resource/deploy/source.go`	What feeds the engine events (real host vs null)
`EvalSource`	`pkg/resource/deploy/source_eval.go`	Drives a real language host
`Step` interface	`pkg/resource/deploy/step.go`	Same / Create / Update / Delete / Replace / Refresh / Read / Import
`stepGenerator`	`pkg/resource/deploy/step_generator.go`	Old × new → steps
`stepExecutor`	`pkg/resource/deploy/step_executor.go`	Schedules + runs steps
`Snapshot`	`sdk/go/common/resource/snapshot.go`	The state document
`Journal`	`pkg/backend/journal/` (interfaces) + `pkg/engine/journal_snapshot.go` (writer)	Crash-safe snapshot write-ahead
`Plugin host`	`pkg/engine/plugin_host.go`, `pkg/engine/plugins.go`	gRPC client to language hosts and providers

Event model

The engine publishes events through pkg/engine/events.go. Subscribers include the CLI display, log streams to Pulumi Cloud, and the policy/analyzer layer.

Event categories:

PreludeEvent — operation start.
ResourcePreEvent / ResourceOutputsEvent / ResourceOperationFailedEvent — per-resource lifecycle.
DiagEvent — log/error from the user program or a plugin.
SummaryEvent — operation summary at the end.
StdoutEventPayload, CancelEvent, PolicyEvent, ...

The display layer treats events as the authoritative source of truth — do not reach into engine state from display code.

Snapshot journaling

pkg/engine/journal_snapshot.go and the Journal interface (used by the backend) implement write-ahead logging for snapshots: every step append/mutation is first journaled, then the snapshot is rewritten. A crash mid-update can be recovered from the journal — see also pkg/backend/journal/.

Plans

pkg/resource/deploy/plan.go implements constrained planning: the engine produces a Plan during preview, the user inspects/persists it, and a subsequent pulumi up --plan=<file> executes only the steps listed (rejecting any deviation). This is the basis of gated deployments.

Lifecycle tests

pkg/engine/lifecycletest/ is a property-based test harness. It generates random Pulumi programs and runs them against pkg/resource/deploy/deploytest/ (a mock provider), asserting that the engine's state machine never enters an invalid state. Default LIFECYCLE_TEST_FUZZ_CHECKS=10000.

When you change anything in pkg/resource/deploy/, run lifecycle tests:

cd pkg && go test -count=1 -tags all ./engine/lifecycletest/...

Entry points for modification

Adding a new operation (rare) — model after pkg/engine/refresh.go. Hook into the CLI from pkg/cmd/pulumi/.
Changing how a step is computed — pkg/resource/deploy/step_generator.go. Always add lifecycle test coverage.
Changing what a step does — pkg/resource/deploy/step.go is the state machine.
Changing event emission — pkg/engine/events.go. New event types are easy to miss in subscribers; check the display layer (pkg/backend/display/) and the cloud event sink.
Changing snapshot persistence — touches both pkg/engine/journal_snapshot.go (writer) and the chosen backend.