Testing

Test count, test variety, and test infrastructure are all unusually large in this repository. The compiler's correctness is the main quality bar, so tests are first-class — every diagnostic, every codegen behavior, every IR transformation has a test fixture, and several test runners are auto-generated from testData/ directories.

The shape of the test suite

There are several distinct test families, each with its own runner conventions:

Family	Where	Purpose
Compiler tests	compiler/tests/, compiler/tests-common/, compiler/tests-common-new/	The big tent: front-end resolution, IR construction, codegen, diagnostics.
FIR tests	compiler/fir/analysis-tests/, compiler/fir/fir2ir/test/	K2 frontend and FIR-to-IR conversion.
JS tests	js/js.tests/	JavaScript backend output and runtime behavior.
Wasm tests	wasm/wasm.tests/	WebAssembly backend.
Native tests	native/native.tests/, kotlin-native/backend.native/tests/	Kotlin/Native end-to-end tests.
Analysis API tests	analysis/*/tests, analysis/analysis-test-framework/	IDE-facing API tests using the FIR backend.
Stdlib tests	libraries/stdlib/test/	Kotlin standard library tests, run on every target.
KGP integration tests	libraries/tools/kotlin-gradle-plugin-integration-tests/	Real-Gradle integration tests. Slow.
JPS tests	jps/, compiler/incremental-compilation-impl/	IntelliJ-side incremental build tests.
Spec tests	compiler/tests-spec/	Tests derived from the Kotlin language specification.

Generated test runners

Most JUnit test classes in this repository are generated from a corresponding testData/ directory. The pattern is:

compiler/testData/codegen/box/numbers/longCompare.kt          ← test input
compiler/tests-gen/.../BlackBoxCodegenTestGenerated.java       ← generated runner

The runner enumerates every test data file in a directory and emits a @Test method per file. To regenerate:

./gradlew generateTests

This is mandatory after adding, removing, or renaming a testData file. Generators live in generators/test-generator/. Per-area generators run from generators/ plus subsystem-specific generators under each area (e.g., compiler/fir/checkers/checkers-component-generator/).

Do not edit *Generated.java files — they will be overwritten on the next regeneration. Edit the underlying testData instead.

Test data format

Test data files are usually plain .kt source with optional directives (special comments) at the top:

• // FILE: foo.kt — split a multi-file test
• // !LANGUAGE: +SomeFeature — set the language version
• // IGNORE_BACKEND: JS, NATIVE — skip on certain backends
• // FIR_IDENTICAL — assert that the K1 and K2 outputs match
• // WITH_STDLIB, // WITH_REFLECT, // WITH_RUNTIME — pull in extra modules
• <!ERROR_NAME!>code<!> markers in diagnostic tests — assert that a diagnostic is or is not reported on a span

For FIR analysis tests specifically, see compiler/fir/analysis-tests/AGENTS.md. The full directive vocabulary is documented in compiler/test-infrastructure/.

Running individual tests

The standard pattern:

./gradlew :compiler:test --tests "org.jetbrains.kotlin.codegen.BlackBoxCodegenTestGenerated" -q

./gradlew :compiler:test --tests "org.jetbrains.kotlin.codegen.BlackBoxCodegenTestGenerated.testSomeTest"

Use -q (quiet) to suppress Gradle's per-task output noise. For FIR tests, the path is usually compiler:fir:<sub-project>:test:

./gradlew :compiler:fir:fir2ir:test --tests "org.jetbrains.kotlin.test.runners.ir.FirLightTreeJvmIrTextTestGenerated"

Updating snapshot test data

Many tests compare actual compiler output to a stored expected file (e.g., .txt IR dumps next to the .kt input). When the expected output changes legitimately, update with:

./gradlew :compiler:test --tests "TestClassName" -Pkotlin.test.update.test.data=true --continue

The flag tells the test framework to write the actual output back to the expected file. Always inspect the resulting diff before committing.

Specialized area tooling

Several areas have their own test conventions and runners — read the area's AGENTS.md first:

• Analysis API — analysis/AGENTS.md. Uses analysis/analysis-test-framework/ and analysis/test-data-manager/.
• Build Tools API — compiler/build-tools/AGENTS.md. Has compatibility tests against multiple compiler versions.
• KGP integration — libraries/tools/kotlin-gradle-plugin-integration-tests/AGENTS.md. Spins up real Gradle builds; slow.
• FIR analysis tests — compiler/fir/analysis-tests/AGENTS.md. Detailed directive reference.
• PSI — compiler/psi/AGENTS.md.

Mute/unmute

Flaky or temporarily-broken tests are muted via compiler/tests-mutes/. Adding to the mute list is acceptable as a temporary measure but should always be paired with a YouTrack ticket.

Test infrastructure modules

If you need to add a new test runner shape, the relevant infrastructure lives in:

• compiler/test-infrastructure/ — base classes, services, runners
• compiler/test-infrastructure-utils/ — utilities (file walking, directive parsing, dump comparison)
• compiler/tests-common-new/ — modern shared base classes for compiler tests
• compiler/test-engine-sandbox/ — JUnit Platform Engine integration

Speeding up local test runs

• Build once, test many times: ./gradlew :compiler:dist produces dist/kotlinc/, used by some test fixtures.
• Use --tests filtering aggressively. Running an entire BlackBoxCodegenTestGenerated is slow.
• The Gradle build cache (~/.gradle/caches) is your friend. Don't clean unless you need to.
• Run generation tasks (generateTests) only when adding or removing test data — they aren't free.