pulumi/pulumi

Design decisions

This page collects the why behind structural choices in pulumi/pulumi. Most of these are decade-old commitments at this point — overturning them would be expensive.

Why multi-process gRPC?

Pulumi's components — CLI, engine, language hosts, providers, analyzers — communicate over gRPC. The alternative would be a monolithic engine that links the language runtimes and provider code directly.

Reasons to choose gRPC:

Polyglot. Providers can be written in any language (most are Go, but TypeScript and Python providers exist). Forcing Go-only would have been a much smaller ecosystem.
Independent versioning. A provider can ship a new version without the user upgrading the CLI, and vice versa. Schema additions are backward-compatible by construction.
Sandboxing. A crashed provider doesn't kill the engine. A user-program panic doesn't corrupt engine state.
Composable. New plugin kinds (analyzers, converters, tools) drop in without touching the engine.

Costs paid:

Process startup is the dominant overhead in fast operations like preview-no-changes.
Debugging is harder when state crosses three processes.
Wire-format changes have to stay backwards compatible forever.

Why schemas?

Provider schemas (pkg/codegen/schema/) drive everything: SDK generation, pulumi convert, IDE completions, the registry. Once a provider has a schema, all downstream tooling falls out for free.

Alternative considered: hand-written SDKs per language per provider (which is roughly what Terraform did historically). Pulumi's bet was that maintaining 1 schema generator + N language emitters would scale better than maintaining N×M hand-written SDKs.

The bet has paid off — adding a new language is now bounded by writing one emitter and one host, not by porting every provider.

Why HCL2 for the IR?

PCL is built on HCL2, the same syntax Terraform uses. This was deliberate: when pulumi convert reads Terraform, it stays in HCL syntax through the bind step before emitting target-language code. Cuts out an entire parsing/transformation layer.

It also means PCL can be hand-written if someone wants — though we've never seen this in practice.

Why split engine and CLI?

pkg/engine/ is the engine; pkg/cmd/pulumi/ is the CLI; the only thing tying them together is pkg/cmd/pulumi/operations/. The split exists so that:

Automation API can drive the engine without going through the CLI surface.
Pulumi Cloud can run the engine server-side (managed deployments).
Tests can run the engine in-process.

The CLI is "just" the most common driver.

Why a separate Workspace abstraction?

sdk/go/common/workspace/ knows about Pulumi.yaml, Pulumi.<stack>.yaml, lockfiles, plugin pin lists, project layout. Originally these all lived in the CLI; the workspace package was carved out so:

Automation API can construct workspaces without files.
Tests can synthesize workspaces.
Plugin install logic (separate from CLI) can read project-pinned versions.

Why immutable resource state?

State is treated as immutable — every mutation produces a new struct via pkg/resource/deploy/state_builder.go. Reasons:

Safety. Goroutines reading old state during step execution don't see partial writes.
Snapshot integrity. Validators run on every saved snapshot; immutability keeps them valid.
Time-travel debugging. The journal can replay old states without restoring mutations.

Why generated `Output<T>` types in Go?

The 480 KB sdk/go/pulumi/types_builtins.go looks excessive but exists because Go's type system (pre-generics) couldn't express Output<T> parametrically. Every primitive needed its own concrete type to provide compile-time safety.

sdk/go/pulumix/Output[T] is the modern, generics-based version. It coexists with the legacy types because removing the legacy types would break every provider SDK.

Why default providers?

When the user writes new aws.s3.Bucket("logs") without supplying a Provider, the engine synthesizes one (a "default provider"). The alternative — forcing explicit providers everywhere — was rejected because:

It made first-time experiences ugly.
95% of users have one provider config per package.
Explicit providers are still possible (and recommended) for advanced use.

Default providers are real resources in the snapshot (pulumi:providers:aws::default_6_10_0). They obey diff/replace semantics like any other resource.

Why journaled snapshots?

pkg/engine/journal_snapshot.go writes a write-ahead log of step results. Without it, a CLI crash mid-update could corrupt the snapshot. With it, the next CLI invocation can recover by replaying the journal.

The Cloud backend has its own server-side journal (pkg/backend/httpstate/journal/) for the same reason.

Why is `step_generator.go` 137 KB?

This file is the diff engine. The complexity comes from:

Multiple step types (Same, Create, Update, Delete, Replace, Refresh, Read, Import, ...) each with subtle conditions.
Replacement modes: Create-before-Replace vs Delete-before-Replace, propagation through dependents.
Targeted updates (--target) prune the dependency graph.
Aliases let the engine recognize renamed resources.
Continue-on-error mode collects failures.
Plan-validation mode constrains step emission.

It is a pure function (modulo provider RPCs), and it has extensive lifecycle test coverage. We've considered splitting it; every split tried so far made the resulting graph harder to reason about.

Why cobra?

pkg/cmd/pulumi/pulumi.go builds a giant cobra command tree. Cobra is the standard Go CLI library and its tree-of-commands model maps cleanly to pulumi <verb> <noun>. The 33 KB of pulumi.go is mostly registration boilerplate.

Why DIY and httpstate?

Pulumi's commercial offering is the cloud backend. Having the DIY backend (filesystem / S3 / GCS / Azure Blob) ensures:

Open-source users have a fully featured product without paying.
Pulumi can self-host development without depending on production cloud.
Air-gapped environments are supported.

The two backends share the Backend interface and are never special-cased in higher layers.

Why a separate Automation API?

sdk/go/auto/, sdk/nodejs/automation/, sdk/python/lib/pulumi/automation/: a typed library to drive Pulumi from inside another program. The alternative — shell out to pulumi — was rejected because:

Parsing CLI stdout is fragile.
Typed outputs are useful in higher-level orchestration.
The CLI flag surface changes; codegen from tools/automation/specification.json keeps the API in sync.

Why renovate over dependabot?

renovate.json5 controls dependency bumps. Renovate was chosen for finer-grained scheduling (groups, batched updates) and for its native support for non-npm/non-Go ecosystems (Python, .NET).

Why mise?

Tool versions vary across contributors — without pinning, "works on my machine" is the norm. .mise.toml pins everything: Go, Node, Python, protoc, golangci-lint, gofumpt, etc. mise is preferred over asdf because of its faster install speed and broader Go-tools support.