neondatabase/neon

Testing

Neon has three testing surfaces: Rust unit tests, Python integration tests, and simulation tests built on the desim deterministic simulator. Each catches a different class of bug, and most non-trivial changes touch at least the first two.

Rust unit tests

Use cargo nextest, not cargo test. Some crates rely on nextest's per-test isolation and won't pass under plain cargo test.

cargo install cargo-nextest --locked
cargo nextest run                           # whole workspace
cargo nextest run -p pageserver             # one crate
cargo nextest run -p pageserver -E 'test(layer_map)'

Nextest is also what CI uses (.github/workflows/_build-and-test-locally.yml). Adding a new test is just #[test] or #[tokio::test] next to the code; conventions:

Prefer tokio::test with (flavor = "multi_thread") for tests that exercise concurrent code.
Use test-log to capture tracing output when a test fails. Many crates already pull it in.
For tests that need a temporary directory, use camino-tempfile (already in Cargo.toml as a workspace dep).
For HTTP/postgres integration in unit tests, look at how pageserver/src/tenant.rs builds harness fixtures and mimic them.

Python integration tests

These live in test_runner/ and drive a real Neon stack — neon_local spins up a pageserver, safekeeper, broker, and one or more compute nodes, runs SQL through them, and verifies behavior.

Setup (one-time):

./scripts/pysync             # installs deps into a poetry venv
poetry shell                 # activate the venv

Run the suite (defaults: debug + release × all four Postgres versions):

CARGO_BUILD_FLAGS="--features=testing" make
./scripts/pytest

Run a single permutation, e.g. release + Postgres 17:

DEFAULT_PG_VERSION=17 BUILD_TYPE=release ./scripts/pytest test_runner/regress/test_branching.py::test_branch_creation

Useful flags:

-x — stop at first failure.
-k <expr> — run tests whose name matches the expression.
--basetemp=/tmp/pytest — keep the working directory on a fast filesystem.
--preserve-database-files — when a test fails, leave the .neon directory around so you can cargo neon against it manually.

Test fixtures live in test_runner/fixtures/. The two most important ones are neon_fixtures.NeonEnv (a full local cluster) and neon_fixtures.PgBin (a thin Postgres wrapper). See test_runner/README.md for the full menu of fixtures.

Performance and load tests

test_runner/performance/ and test_runner/pg_clients/ hold benchmark-style tests that run nightly in CI (.github/workflows/benchmarking.yml, proxy-benchmark.yml, large_oltp_benchmark.yml). They are not part of the default ./scripts/pytest run.

pageserver/pagebench/ is a Rust-based benchmarking tool that drives the page service directly and is invoked from .github/workflows/periodic_pagebench.yml.

Simulation tests

libs/desim/ is a small deterministic simulator. The safekeeper has the most extensive simulation tests — see safekeeper/tests/walproposer_sim/ and safekeeper/spec/ (TLA+ specifications). The tests use a virtual clock and synchronized network to exercise consensus edge cases that would be hard to reproduce on real hardware.

Run them like ordinary unit tests:

cargo nextest run -p safekeeper -E 'test(walproposer_sim)'

Failure traces include simulated wall-clock and a deterministic seed so they can be re-run identically.

Property and randomized tests

Some crates use proptest or rstest-driven parametric tests. Search for #[rstest] or proptest! to find them. They are run by the same cargo nextest command as the rest of the unit tests.

Sanitizer builds

CI runs sanitizer builds (.github/workflows/build_and_test_with_sanitizers.yml) periodically. Locally:

WITH_SANITIZERS=yes BUILD_TYPE=release make
WITH_SANITIZERS=yes ./scripts/pytest

These are slower; reach for them when triaging a memory-safety smell.

Test-only feature flag

Many crates gate test-only code behind --features testing. Flip it on for builds that integration tests need: make automatically passes CARGO_BUILD_FLAGS="--features=testing" when you set the env var. Notable users:

pageserver — exposes failpoints under --features testing.
proxy — enables local_proxy and a few in-memory auth backends.

Failpoints

Many subsystems are instrumented with the fail crate. CI tests trigger known failure points to verify recovery paths. To inject a failpoint manually during a Python integration test:

env.pageserver.allowed_errors.append(".*injected.*")
env.pageserver.http_client().configure_failpoints([("name-of-failpoint", "return")])

See pageserver/src/tenant.rs and friends for the names available.

CI matrix at a glance

The two big workflows you should be aware of:

Workflow	What it does	When
`.github/workflows/build_and_test.yml`	Build + Rust unit tests + Python integration tests on Linux	Every push and PR
`.github/workflows/build_and_test_fully.yml`	Same plus all Postgres versions and a slower sanitizer pass	On request via label

If a test is flaky, mark it with @pytest.mark.flaky rather than skipping it outright; flakiness is tracked.