Testing

The Swift toolchain has one of the largest test suites in any compiler. This page surveys the test layout, the runners, and how to write new tests.

Layout

Directory	Purpose
test/	Per-feature compiler/runtime tests; ~16,489 files, the workhorse suite.
validation-test/	Slower, deeper validation tests; ~5,600 files. Run on CI but not local fast loops.
unittests/	C++ Google-Test unit tests for compiler libraries.
benchmark/	Micro-benchmarks for the standard library and codegen quality.
test-tools/	Tooling-specific tests under SourceKit, etc.

test/ is split by area:

• test/Sema/, test/Constraints/ -- type-checker tests.
• test/Parse/ -- parser tests.
• test/SILGen/, test/SIL/, test/SILOptimizer/ -- SIL pipeline tests.
• test/IRGen/ -- LLVM IR-level codegen tests.
• test/stdlib/ -- standard library behavioral tests.
• test/Concurrency/, test/Generics/, test/Macros/, test/Interop/, test/embedded/ -- feature areas.

Test runner: lit

The compiler test suite uses LLVM's lit. Each test file is a comment block instructing lit how to compile / run it. Example header:

// RUN: %target-typecheck-verify-swift -enable-experimental-feature TypedThrows

Common substitutions:

• %target-swiftc_driver, %target-swift-frontend -- correctly-tested driver/frontend invocations.
• %target-typecheck-verify-swift -- runs typecheck and verifies inline // expected-error{{...}} annotations.
• %FileCheck -- matches stdout/stderr against // CHECK-... patterns.
• %S, %t, %target-os, %target-cpu -- standard lit substitutions.

The full list of substitutions is in test/lit.cfg.

Running tests

The project ships several wrappers; from fastest local loop to full CI:

# Single file
./bin/swift-frontend -typecheck path/to/test.swift

# A single lit test
./build/.../swift-macosx-arm64/bin/llvm-lit \
    ../../../swift/test/Sema/typecheck_basic.swift -v

# A whole directory
./utils/run-test --build-dir ../build/Ninja-RelWithDebInfoAssert \
    swift/test/Sema

# All swift tests for the host
ninja -C ../build/Ninja-RelWithDebInfoAssert/swift-macosx-arm64 check-swift

# All tests including validation
ninja -C ../build/Ninja-RelWithDebInfoAssert/swift-macosx-arm64 check-swift-all

utils/run-test (utils/run-test) is a Python wrapper that calls lit with the right arguments.

Writing tests

Type-checker / parser

// RUN: %target-typecheck-verify-swift

func foo(_ x: Int) -> String { return "\(x)" }

let _ = foo("hi") // expected-error {{cannot convert value of type 'String' to expected argument type 'Int'}}

SIL

A typical SIL test pre-verifies the IR with FileCheck:

// RUN: %target-swift-frontend -emit-sil -O %s | %FileCheck %s

@inline(never)
public func add(_ x: Int, _ y: Int) -> Int { x + y }

// CHECK-LABEL: sil @$s4main3addyS2i_SitF
// CHECK: integer_literal $Builtin.Int64

Runtime

test/stdlib/ and test/Runtime/ typically use StdlibUnittest from stdlib/private/StdlibUnittest/:

// RUN: %target-run-simple-swift
// REQUIRES: executable_test

import StdlibUnittest

let suite = TestSuite("Foo")
suite.test("bar") { expectEqual(1, 1) }
runAllTests()

IRGen

// RUN: %target-swift-frontend -emit-ir %s | %FileCheck %s

// CHECK: define swiftcc i64 @"$s4main3addyS2i_SitF"
public func add(_ x: Int, _ y: Int) -> Int { x + y }

Stdlib unit tests

stdlib/private/StdlibUnittest/ provides the harness used by everything in test/stdlib/. Important utilities:

• expectEqual, expectTrue, expectCrashLater -- assertions.
• TestSuite and runAllTests() -- test entry points.
• gyb-generated permutations of numeric tests in test/stdlib/.

Benchmarks

Benchmarks live in benchmark/single-source/, benchmark/multi-source/, and benchmark/cxx-source/. The runner is in benchmark/scripts/. To build and run:

./utils/build-script -R --benchmark
./build/.../bin/Benchmark_O --num-iters=3 ArrayInClass

Benchmark naming is documented in benchmark/Naming.md.

Source compatibility suite

For language-level changes, run the source compatibility suite via @swift_ci please test source compatibility. It compiles a corpus of real-world Swift packages.

Common pitfalls

• Forgetting // REQUIRES: lines for tests that need particular targets, OS versions, or features.
• Using // CHECK: patterns that don't anchor; many SIL tests need // CHECK-LABEL: to scope.
• Tests under test/Concurrency/ typically require // REQUIRES: concurrency.
• Tests under test/embedded/ typically require // REQUIRES: embedded and use a custom test driver.