Testing

The canonical command playbook is docs/agents/testing-flow.md. This page summarizes the layers of test infrastructure in the repo and points you at the right one.

Test layers

graph TD
  unit[Unit tests<br/>pkg/**/*_test.go] --> integration[Integration<br/>tests/integrationtest]
  unit --> realtikv[Real TiKV<br/>tests/realtikvtest]
  integration --> bazel[Bazel CI<br/>make bazel_test]
  realtikv --> bazel
  unit --> testkit[testkit framework<br/>pkg/testkit]

Layer	Location	When to use
Unit tests	pkg/<pkg>/*_test.go	Pure logic, single package. Default.
Testkit-backed unit tests	use pkg/testkit/ to spin a TiDB on unistore in-process	Multi-package SQL behaviour without a real cluster.
Integration tests	tests/integrationtest/ (inputs in t/, golden output in r/)	SQL-level behaviour, golden testing.
RealTiKV tests	tests/realtikvtest/	Behaviour that depends on real TiKV/PD.
Cluster integration	tests/clusterintegrationtest/, tests/graceshutdown/, tests/globalkilltest/, tests/readonlytest/	Multi-node scenarios, graceful shutdown, kill, read-only mode.
Compatibility	br/compatibility/	Cross-version compatibility for BR.

Running unit tests

The default invocation:

go test -run TestX -tags=intest,deadlock ./pkg/<pkg>/...

Always use -tags=intest,deadlock (some assertions are gated on intest; deadlock enables sync.Mutex deadlock detection). Prefer targeted runs.

Failpoint-using tests

Many tests inject faults via github.com/pingcap/failpoint. The decision rule (from AGENTS.md):

rg --fixed-strings "failpoint." pkg/<pkg>
rg --fixed-strings "testfailpoint." pkg/<pkg>
test -f pkg/<pkg>/BUILD.bazel && rg --fixed-strings "@com_github_pingcap_failpoint//:failpoint" pkg/<pkg>/BUILD.bazel

If any of those finds a match, run with the helper script:

./tools/check/failpoint-go-test.sh pkg/<pkg> -run TestX

The script enables failpoints, runs go test, and disables them on exit. Don't run plain go test on a failpoint-using package — symbols won't be linked properly and tests will fail in confusing ways. The dedicated tidb-failpoint-test-runner skill walks through the decision.

Integration tests

pushd tests/integrationtest
./run-tests.sh -r <name>      # runs and records a single test
popd

Inputs are in tests/integrationtest/t/<area>/<name>.test; expected outputs in tests/integrationtest/r/<area>/<name>.result. After a successful recording, review the diff in r/ carefully — tests/integrationtest/run-tests.sh is the integration-test recorder, distinct from the -record flag accepted by certain unit-test suites.

The tidb-integrationtest-recorder skill captures the verification rules for these recordings.

RealTiKV tests

These need a TiUP playground. The recommended template:

PD_ADDR=127.0.0.1:2379
(
  cleanup() {
    [ -n "${PLAYGROUND_PID:-}" ] && kill "${PLAYGROUND_PID}" 2>/dev/null || true
    [ -n "${PLAYGROUND_PID:-}" ] && wait "${PLAYGROUND_PID}" 2>/dev/null || true
    rm -rf "${HOME}/.tiup/data/realtikvtest"
  }
  trap cleanup EXIT INT TERM

  tiup playground --mode tikv-slim --tag realtikvtest &
  PLAYGROUND_PID=$!

  until curl -sf "http://${PD_ADDR}/pd/api/v1/version" >/dev/null; do sleep 1; done
  go test -run TestX -tags=intest,deadlock ./tests/realtikvtest/<dir>/...
)
! curl -sf "http://${PD_ADDR}/pd/api/v1/version"

Cleanup is mandatory: leftover TiUP data confuses subsequent runs. The tidb-realtikv-runner skill encodes this lifecycle.

Bazel CI

CI runs the equivalent of:

make bazel_prepare    # if any of the trigger conditions matched
make bazel_test       # the full suite

Plus subsystem-specific targets (bazel_brietest, bazel_pessimistictest, bazel_addindextest1..4, bazel_importintotest1..4, etc.). Locally you rarely need these — run a targeted go test first.

Bug fixes

A bug-fix PR must include a regression test that fails before the fix and passes after. The rule is in AGENTS.md -> Quick Decision Matrix (Bug fix). When recording golden output for the test, verify the diff is minimal and only touches the regression target.

Triaging unexpected diffs

If your test sees plan/result/testdata diffs that look unrelated to your change, read docs/agents/tidb-test-diff-triage (also exposed as the tidb-test-diff-triage skill). Common causes are missing failpoint enablement, stale bazel_prepare state, or shared globals in test helpers.