Test harness

The shared infrastructure that loads test fixtures, runs the compiler against them, captures output, and compares against expected baselines. Powers all four test runner modes (compiler, conformance, fourslash, project) plus the in-process unit tests.

Source

Directory	Role
src/harness/	Test runtime: virtual filesystem, fake hosts, fourslash interpreter, IO helpers
src/testRunner/	Mocha runners; one per test category
tests/cases/	Test inputs (~19,000 files)
tests/baselines/reference/	Expected outputs
tests/baselines/local/	Generated outputs (gitignored)
scripts/build/tests.mjs	hereby runtests orchestration

Purpose

Most TypeScript tests are baseline tests: a .ts file is compiled, the resulting .js / .d.ts / errors / types / symbols / source maps are captured, and the captures are diffed against a stored baseline. A test fails if any baseline differs. To accept new outputs intentionally, the workflow is:

hereby runtests --tests=mytest    # generate local baselines
git diff tests/baselines           # inspect changes
hereby baseline-accept             # promote local → reference

This forces every behavioural change to be visible in review.

Key abstractions

Symbol	File	Role
compileFiles, compileDeclarations	src/harness/compilerImpl.ts	Run the compiler in-process against test inputs
vfs.FileSystem	src/harness/vfsUtil.ts	In-memory case-sensitive/insensitive filesystem
fakes.CompilerHost, fakes.ParseConfigHost, fakes.System	src/harness/fakesHosts.ts	CompilerHost impls backed by vfs.FileSystem
Harness.Compiler	src/harness/harnessIO.ts	Coordinates one test compilation + baseline capture
IO, Harness.IO	src/harness/harnessIO.ts	Filesystem abstraction used everywhere in the harness
Baseline	src/harness/harnessIO.ts	Compare generated text against tests/baselines/reference/
FourSlash.runFourSlashTest	src/harness/fourslashImpl.ts	The 5,200-line fourslash interpreter
Test runners	src/testRunner/*Runner.ts	One per category — compilerRunner, fourslashRunner, projectsRunner, transpileRunner

How it works

graph TD
    Mocha["mocha (runtests)"] --> Runners["Runner.runTests"]
    Runners --> Disco["enumerate tests/cases/<dir>"]
    Disco --> Each["for each test"]
    Each --> Parse["parse compiler directives (// @target, // @Filename, ...)"]
    Parse --> VFS["build vfs.FileSystem"]
    VFS --> Host["fakes.CompilerHost"]
    Host --> Compile["createProgram + emit"]
    Compile --> Capture["capture .js / .d.ts / errors / types / symbols / source maps"]
    Capture --> Baseline["Baseline.run vs tests/baselines/reference"]
    Baseline --> Result["pass/fail + tests/baselines/local diff"]

Test categories

Runner	Inputs	What's checked
compiler	tests/cases/compiler/*.ts	.js, .d.ts, .errors.txt, .types, .symbols, optionally .js.map
conformance	tests/cases/conformance/**/*.ts	Same as compiler
fourslash	tests/cases/fourslash/*.ts	Language-service queries via verify.* assertions
project	tests/cases/project/*	Multi-file projects with tsconfig.json
projects	tests/cases/projects/*	Larger-scale project-reference scenarios
transpile	tests/cases/transpile/*.ts	transpileModule output
unittests	src/testRunner/unittests/**/*.ts + tests/cases/unittests/**	Direct mocha tests for utilities

Compiler-test directives

A compiler test is a normal .ts file with leading comments:

// @Filename: b.ts
import { x } from './a';

// @strict: true
// @target: ES2015
// @lib: ES2015,DOM
// @Filename: a.ts
export const x = 1;

Each // @<name>: <value> directive maps to a CompilerOptions field. // @Filename: introduces a virtual file in the test's vfs.

Fourslash

Fourslash tests live in tests/cases/fourslash/. They use a unique syntax:

/// <reference path="fourslash.ts" />

////function foo(/*1*/x: number) {
////    return /*2*/x + 1;
////}

goTo.marker('1');
verify.quickInfoIs('(parameter) x: number');

//// lines are the source. /*name*/ are markers. verify.* assertions are interpreted by the fourslash runtime in src/harness/fourslashImpl.ts which constructs a real LanguageService and runs queries against it.

VFS

src/harness/vfsUtil.ts implements an in-memory filesystem with copy-on-write semantics, configurable case sensitivity, and unionFs-style overlays. Tests can build elaborate filesystems (multiple node_modules, symlinks, case differences) without touching disk.

Parallel execution

hereby runtests-parallel shards tests across worker processes via src/testRunner/parallel/. Each worker runs an isolated mocha and reports back to a coordinator.

Coverage

hereby runtests --coverage runs under c8 to produce LCov / monocart reports. The .c8rc.json at the repo root configures it.

Integration points

• Built by hereby tests into built/local/run.js (mocha entry point).
• Reads test fixtures via IO/vfs.FileSystem.
• Calls into the public compiler API exactly as users do.
• Writes baselines to tests/baselines/local/.
• Comparison logic lives in Harness.Baseline.

Entry points for modification

• Adding a new compiler test: drop a .ts file under tests/cases/compiler/ (or appropriate tests/cases/conformance/ subdirectory), run hereby runtests --tests=<name>, accept baselines.
• Adding a fourslash test: drop a .ts file under tests/cases/fourslash/, follow the syntax above.
• Adding a runner mode: subclass RunnerBase in src/testRunner/runner.ts.
• Changing baseline comparison: src/harness/harnessIO.ts.

See how-to-contribute/testing for the contributor-side workflow.