Testing

Both products in this repo treat tests as load-bearing artifacts: lint rules, formatter output, and type-checker behavior are pinned by snapshots that are read in code review.

Test runners

cargo nextest run is recommended over cargo test for speed and better output. The canonical environment from AGENTS.md:

CARGO_PROFILE_DEV_OPT_LEVEL=1 \
INSTA_FORCE_PASS=1 INSTA_UPDATE=always \
CARGO_PROFILE_DEV_DEBUG="line-tables-only" \
MDTEST_UPDATE_SNAPSHOTS=1 \
cargo nextest run

What each variable does:

Variable	Effect
CARGO_PROFILE_DEV_OPT_LEVEL=1	Optimization level 1 in dev — much faster than 0 for the test suite.
CARGO_PROFILE_DEV_DEBUG="line-tables-only"	Keeps backtraces useful while shrinking debug info.
INSTA_FORCE_PASS=1	Run all snapshot tests even when some change.
INSTA_UPDATE=always	Update mismatched snapshots on disk.
MDTEST_UPDATE_SNAPSHOTS=1	Same idea for ty's mdtest harness.

After the run, review the snapshot diffs. cargo insta review is interactive; cargo insta accept accepts all.

Test taxonomy

Kind	Crate	Where it lives
Snapshot rule tests	ruff_linter	crates/ruff_linter/src/rules/<plugin>/mod.rs driving fixtures under crates/ruff_linter/resources/test/fixtures/<plugin>/ and snapshots under crates/ruff_linter/src/rules/<plugin>/snapshots/
Formatter snapshots	ruff_python_formatter	crates/ruff_python_formatter/tests/ with fixtures under resources/test/fixtures/
Parser tests	ruff_python_parser	crates/ruff_python_parser/resources/ and inline tests
ty mdtest	ty_python_semantic	crates/ty_python_semantic/resources/mdtest/**/*.md
ty unit tests	ty_python_semantic	inline #[test] and crates/ty_python_semantic/tests/
LSP tests	ruff_server, ty_server	inline + tests/
WASM smoke	ruff_wasm, ty_wasm	wasm-pack runners (run separately)
Fuzz	fuzz/	cargo fuzz run <target>

Insta snapshot tests

Most rule tests follow the same pattern:

#[test]
fn unused_imports() {
    let diagnostics = test_path(
        Path::new("pyflakes/F401.py"),
        &settings::LinterSettings::for_rule(Rule::UnusedImport),
    ).unwrap();
    assert_messages!(diagnostics);
}

assert_messages! writes to crates/ruff_linter/src/rules/pyflakes/snapshots/ruff_linter__rules__pyflakes__tests__unused_imports.snap. When the snapshot is empty or missing, INSTA_UPDATE=always populates it. Always commit the new snapshot alongside the change.

mdtest (ty)

mdtest files are Markdown documents with embedded fenced Python code and a structured DSL for declaring expected diagnostics. They serve double duty: they're tests and readable documentation of how the type checker behaves.

A minimal example (crates/ty_python_semantic/resources/mdtest/some-feature.md):

# Some feature

```py
x: int = "hello"  # error: [invalid-assignment] "Cannot assign str to int"
```

To run a single file:

cargo nextest run -p ty_python_semantic -- mdtest::<path/to/some-feature.md>

To filter inside a file by header text:

MDTEST_TEST_FILTER="header substring" cargo nextest run -p ty_python_semantic -- mdtest::<file.md>

The harness is crates/mdtest. See its README for the full DSL: section nesting, multi-file projects, expected error codes, type-of assertions, and <rev block annotations.

Fixtures

Lint and formatter fixtures are real Python files. Conventions:

• One fixture per rule when possible. Multiple rules in one fixture is fine when they overlap.
• Fixtures should exercise both the trigger and non-trigger cases. Comments at the top describe what to expect.
• For autofixes, the fixture file is the input; the snapshot captures the diagnostic and the post-fix output.

Fuzzing

The fuzz/ directory contains cargo-fuzz targets, including:

• A formatter idempotency fuzzer (format twice, expect identical output).
• A parser fuzzer (no panics on arbitrary bytes).

Run a target with:

cargo +nightly fuzz run <target> -- -max_total_time=60

Coverage and corpus runs

There is no formal line-coverage gate, but two corpus-style harnesses are run periodically:

• Black-compatibility check — runs the formatter against a corpus of public projects and diffs against Black.
• ecosystem analyzer (ty) — runs ty against a curated set of real-world projects and reports diagnostic deltas. See scripts/setup_primer_project.py to reproduce a project locally.

When tests fail in CI but pass locally

• Default to running with the canonical env vars at the top of this page.
• Re-run uvx prek run -a to catch generated-file drift.
• If RUFF_UPDATE_SCHEMA=1 cargo test shows a diff, commit the regenerated schema.
• For ty, ensure MDTEST_UPDATE_SNAPSHOTS=1 was set when you regenerated mdtest output.