Testing

LLVM has two test layers in active use across every subproject: lit-driven regression tests and Google-Test-based unit tests. They cover different things and have different ergonomics.

Lit regression tests

lit (llvm/utils/lit/) is the LLVM Integrated Tester. It's a Python package that walks a directory of test files, extracts RUN: directives from the comments at the top of each test, runs them as shell commands, and reports pass/fail.

A typical lit test (llvm/test/CodeGen/X86/popcnt.ll) looks like:

; RUN: llc -mtriple=x86_64-- -mattr=+popcnt < %s | FileCheck %s

define i32 @popcount32(i32 %x) {
; CHECK-LABEL: popcount32:
; CHECK: popcntl
  %r = call i32 @llvm.ctpop.i32(i32 %x)
  ret i32 %r
}

declare i32 @llvm.ctpop.i32(i32)

The RUN: line is the command. %s is the test file itself. The output is piped through FileCheck (llvm/utils/FileCheck/), which asserts on patterns embedded in the same file.

Running lit

The build emits a check-<project> target per subproject:

ninja -C build check-llvm     # llvm/test/
ninja -C build check-clang    # clang/test/
ninja -C build check-mlir     # mlir/test/
ninja -C build check-lld
ninja -C build check-lldb
ninja -C build check-flang
ninja -C build check-cxx      # libc++ tests
ninja -C build check-all      # everything that's enabled

To run a single file or directory:

build/bin/llvm-lit -v llvm/test/CodeGen/X86/popcnt.ll
build/bin/llvm-lit -v llvm/test/Transforms/InstCombine/

Useful flags:

• -v — verbose; shows the failing command and FileCheck diff
• -a — always show output, even for passing tests
• -j N — set parallelism
• --filter <regex> — run only tests matching a regex
• --time-tests — print per-test timings

FileCheck patterns

FileCheck directives in the test file:

• ; CHECK: — match this substring on some output line
• ; CHECK-NEXT: — match on the next output line
• ; CHECK-NOT: — assert this pattern does not appear before the next CHECK
• ; CHECK-LABEL: — match a label and reset the search anchor (used to delimit functions in IR/asm output)
• ; CHECK-DAG: — match in any order
• ; CHECK-COUNT-N: — match exactly N occurrences
• [[NAME:regex]] — capture for cross-reference
• [[NAME]] — back-reference

FileCheck is also a usable C++ library at llvm/lib/FileCheck/.

Updating golden output

Some tests are auto-generated from the compiler's actual output. If a benign codegen change shifts the output, you can regenerate the expectations rather than hand-edit them:

llvm/utils/update_test_checks.py llvm/test/Transforms/InstCombine/and-icmp.ll
llvm/utils/update_llc_test_checks.py llvm/test/CodeGen/X86/popcnt.ll
llvm/utils/update_mir_test_checks.py llvm/test/CodeGen/AArch64/foo.mir

The full set of updaters lives in llvm/utils/. Each picks a different output convention.

Unit tests (Google Test)

Each subproject has a unittests/ directory with C++ test code linked against Google Test (third-party/unittest/ bundles a copy). Examples:

• llvm/unittests/Support/
• clang/unittests/AST/
• mlir/unittests/Dialect/

Build and run:

ninja -C build SupportTests
build/unittests/Support/SupportTests

The check-llvm-unit target runs all LLVM-core unit tests; check-<project>-unit does the same per subproject. check-<project> runs both lit and unit tests.

Filter individual unit tests by Google-Test syntax:

build/unittests/Support/SupportTests --gtest_filter='StringRef.Construction'

Other test layers

• llvm/test/tools/ — black-box tests for command-line tools (llvm-objdump, llvm-readobj, etc.).
• llvm/test/MC/ — assembler tests, run via llvm-mc.
• compiler-rt/test/ — sanitizer and profile-runtime tests, often requiring a working host clang.
• libcxx/test/ — the libc++ conformance suite, run via lit.
• cross-project-tests/ — tests that need multiple subprojects (e.g., debug-info round trips between Clang, LLVM, and LLDB).

Pre-commit CI

Pull requests run through buildbots configured in .ci/ and .github/. The CI runs a representative subset of check-* targets per subproject. CI failures are usually copy-pasteable logs you can repro locally.

Test hygiene

• Every behavior change ships with a regression test. The reviewer will say so if you forget.
• Prefer narrow, focused tests over giant integration tests. A new InstCombine fold gets one IR test that exercises just the new fold.
• Use update_*_test_checks.py to regenerate expected output rather than handwriting ; CHECK: lines for codegen tests.
• Don't XFAIL a test to land your patch — fix the test or fix the regression.