compiletest

src/tools/compiletest/ is the test runner for everything under tests/. When you run ./x test tests/ui or ./x test tests/codegen-llvm, this is the program doing the work.

What it does

• Discovers test files in a suite directory
• Parses directives at the top of each test (//@ compile-flags: …)
• Invokes rustc / cargo / a custom command with the right flags
• Compares output against expected .stderr / .stdout files (and diffs on mismatch)
• Supports --bless to overwrite expected outputs
• Filters tests via --test-args=<pattern>
• Runs in parallel across CPUs
• Produces JUnit-style output for CI consumption

Suites

Each suite under tests/ is an independent compiletest invocation with a different mode:

Mode	Suites that use it	What it does
ui	tests/ui, tests/rustdoc-ui	Compile and compare diagnostics
run-make	tests/run-make	Run an rmake.rs driver script
codegen	tests/codegen-llvm	FileCheck-style assertions on LLVM IR
assembly	tests/assembly-llvm	FileCheck on assembly output
mir-opt	tests/mir-opt	Snapshot of MIR after a pass
incremental	tests/incremental	Multi-revision incremental tests
pretty	tests/pretty	Round-trip parsing/pretty-print
coverage	tests/coverage	Source-based coverage
debuginfo	tests/debuginfo	Debugger-driven (LLDB/GDB/CDB)
rustdoc	tests/rustdoc-html	rustdoc HTML output checks
crashes	tests/crashes	Currently-known ICEs (must crash)

Directive format

Directives appear at the top of every test as comments starting with //@. They configure compiletest for that test:

//@ check-pass
//@ edition: 2021
//@ compile-flags: -Zfoo=bar -Cno-prepopulate-passes
//@ ignore-windows
//@ revisions: a b
//@[a] compile-flags: --cfg=a
//@[b] compile-flags: --cfg=b

#![feature(my_feature)]
fn main() {
    let x: u32 = "string"; //~ ERROR mismatched types
}

The full list lives in src/tools/compiletest/src/directive_list.rs. Common ones:

Directive	Effect
check-pass / check-fail	type-check expected to succeed / fail
build-pass / build-fail	compilation through codegen succeeds / fails
run-pass / run-fail	compile + run; success / fail at runtime
edition: <YEAR>	set the edition
compile-flags: <flags>	passed to rustc
revisions: a b c	run multiple compilations with different cfgs
aux-build: foo.rs	compile a dependency crate first
ignore-<platform> / only-<platform>	platform gating
needs-asm-support / needs-unwind / …	capability-gating

Annotations

Inside a UI test, //~ ERROR …, //~ WARN …, etc. (called "annotations") tell compiletest which diagnostics are expected on which line. The exact match is loose — a substring match — but the line and severity must match.

//~ anchors to the same line. //~^ anchors one line above, //~^^ two lines above, etc. //~| continues the previous diagnostic for an additional sub-message.

--bless

Snapshot-based suites (UI, MIR-opt, codegen) regenerate expected output with --bless:

./x test tests/ui/borrowck --bless
./x test tests/mir-opt --bless

After running, review the diffs (git diff -- tests/) before committing.

How directives parse

src/tools/compiletest/src/directives.rs parses each test header. The parser is conservative: unknown directives are warned about, and //@ is a hard syntax error if malformed. The directive list is kept tightly controlled by tidy; adding a new directive needs a corresponding entry in directive_list.rs.

How tests find rustc

compiletest is given a --rustc-path pointing to the stage compiler being tested. Bootstrap passes the right path automatically. When you run ./x test --stage 1 tests/ui, the stage 1 compiler is what gets exercised.

Performance

The full tests/ui/ suite runs ~14,000+ tests. compiletest:

• Forks rustc for each test (no in-process compilation — clean state per test)
• Parallelizes over CPUs
• Caches some setup (aux-builds, doctest dependencies)

Even so, the full suite takes minutes on a fast machine. Filter aggressively in development: ./x test tests/ui/borrowck --test-args borrow.rs runs one file.

Output formats

• Default — colored, terse on success, verbose on failure
• --json — machine-readable for CI
• --summarize — JUnit XML for GitHub Actions display

Run-make tests

tests/run-make/<dir>/rmake.rs is a Rust file that uses run-make-support to drive a multi-step build. compiletest compiles rmake.rs against run-make-support, runs it, and the test passes iff the support library reports success.

This is the escape hatch for "this is too complex for a UI test": multi-crate builds, custom linker invocations, custom file layouts, anything that would need shelling out to a Makefile in another test runner.

Entry points for modification

• New directive → src/tools/compiletest/src/directives.rs + directive_list.rs
• New mode → significant work; touches the runner crate
• Pattern-matching tweaks for diagnostics → the test annotation matcher in compiletest

See also

• Testing — user-facing testing workflow
• src/tools/run-make-support/ — the run-make helper crate
• rustc-dev-guide: testing