pulumi/pulumi

Testing

pulumi/pulumi has multiple distinct test surfaces. Knowing which one applies saves a lot of time.

The four test surfaces

Surface	Location	Run with	Speed
Unit tests (Go)	next to source as `_test.go`	`make test_fast`	seconds
Lifecycle / fuzz tests	`pkg/engine/lifecycletest/`	`cd pkg && go test ./engine/lifecycletest/...`	minutes
Integration tests	`tests/integration/`	`make test_all` (requires built CLI)	tens of minutes
Conformance tests	`tests/integration/...`, driven by `scripts/run-conformance.sh`	`./scripts/run-conformance.sh`	tens of minutes

Unit tests

Standard Go testing. Build tags scope big test suites (-tags all).

make test_fast                                          # everything fast
cd pkg && go test -count=1 -tags all ./codegen/go/...   # one package
cd pkg && go test -run TestSpecific ./engine/...        # one test

Notable conventions:

Golden file tests. Many code-gen tests assert that generated output matches a checked-in fixture. Set PULUMI_ACCEPT=1 to update fixtures: PULUMI_ACCEPT=1 go test ./codegen/go/.... Don't accept naively — eyeball the diff.
Mocks. SDK mocks live at sdk/go/pulumi/mocks.go and sdk/python/lib/pulumi/runtime/mocks.py; the deployment-side mock provider is pkg/resource/deploy/deploytest/.

Lifecycle tests

pkg/engine/lifecycletest/ is a property-based suite that exercises the engine end-to-end against a mock provider. It uses pgregory.net/rapid to generate random programs.

Default Rapid checks per run: LIFECYCLE_TEST_FUZZ_CHECKS=10000 (set in the Makefile).
Override locally for faster iteration: LIFECYCLE_TEST_FUZZ_CHECKS=100 make ....
Failing seeds are reported by Rapid and can be replayed.

If you change anything in pkg/resource/deploy/, run lifecycle tests. They catch correctness regressions that unit tests miss.

tests/integration/ is the heaviest suite. Each test is a small Pulumi program in a real language under tests/integration/<name>/<lang>/. The test driver builds the program, runs pulumi up, asserts on outputs, then pulumi destroys.

Prerequisites:

make build has produced bin/pulumi and the SDKs are installed (make install for Node).
bin/ is on PATH.
For Pulumi Cloud–backed tests, PULUMI_ACCESS_TOKEN is set; otherwise the DIY backend is used.

Run a single integration test:

cd tests
go test -count=1 -tags all -run TestSomething ./integration/...

Failing integration tests usually leave a working directory you can poke at. Look for tmp/... in the test output.

Conformance tests

scripts/run-conformance.sh runs a suite that asserts every supported language gets identical results from the same Pulumi program. This catches divergence between, say, Go-SDK behavior and Python-SDK behavior. The fixtures live under tests/integration/... and are tagged appropriately.

Performance gates

.github/workflows/ci-performance-gate.yml runs tests/performance/ benchmarks and compares against a baseline. If your PR slows things down, expect to be asked about it.

pkg/backend/httpstate/snapshot_benchmark_test.go is the canonical snapshot benchmark for cloud backend perf.

Per-language SDK tests

Each SDK has its own test runner:

Node: cd sdk/nodejs && make test_fast (mocha + biome + eslint).
Python: cd sdk/python && make test_fast (pytest + ruff).
Go: standard go test from sdk/.

How to run just the changed-area tests

This is one of the most useful patterns when iterating:

# changed pkg/engine/...
cd pkg && go test -count=1 -tags all ./engine/...

# changed pkg/resource/deploy/...
cd pkg && go test -count=1 -tags all ./resource/deploy/...

# changed sdk/go/pulumi/...
cd sdk && go test -count=1 ./go/pulumi/...

# changed proto/
make build_proto && make check_proto

Test infrastructure helpers

pkg/resource/deploy/deploytest/ — the mock provider and supporting fixtures used by lifecycle tests.
tests/testprovider/ and tests/testprovider-py/ — built-in test providers used by integration tests.
tests/testutil/ — helpers shared by integration tests.
pkg/testing/, sdk/go/common/testing/ — shared test helpers.

When tests fail in CI but pass locally

The usual causes:

Environment difference: stale bin/pulumi or stale generated proto. Try make build_proto && make build.
Module drift across pkg/, sdk/, tests/. Run make tidy_fix.
Test order dependency. Add -shuffle=on or -count=10 to reproduce.
Plugin caching. Wipe ~/.pulumi/plugins/ if a provider version changed.