Testing

Ruby has more test infrastructure than most languages — partly because behaviour matters across implementations (CRuby, JRuby, TruffleRuby) and partly because thirty years of language history has produced a lot of compatibility surface to defend.

The four test layers

Layer	Directory	Runner	Purpose
Bootstrap test	bootstraptest/	bootstraptest/runner.rb (via make btest)	Smoke tests; runs against a freshly-built miniruby very early in make. Tests core VM mechanics.
Basic test	basictest/	make test-basic	A handful of pass/fail scripts the binary must succeed on before the full suite is even attempted.
Test suite	test/ruby/, test/<gem>/	test-unit (vendored, tool/test/runner.rb)	The main CRuby test corpus. Hundreds of files, thousands of tests.
Behaviour spec	spec/ruby/	MSpec (vendored under spec/mspec/)	The cross-implementation ruby/spec. Documents observable language behaviour, run by JRuby and TruffleRuby too.

A typical run order from a fresh build:

make btest       # ~30s, lots of bytecode/VM tests
make test-basic  # ~5s
make test-all    # 5-15 min depending on hardware
make test-spec   # 5-15 min
make check       # all of the above

make exam adds linting and leak checks on top of make check.

Running one file

make test-all and make test-spec accept a TESTS argument:

make test-all TESTS="test/ruby/test_array.rb"
make test-all TESTS="-n /test_each_with_index/ test/ruby/test_array.rb"
make test-spec MSPECOPT="spec/ruby/core/array/each_spec.rb"

For ad-hoc runs without going through make:

./tool/runruby.rb test/ruby/test_array.rb
./tool/runruby.rb spec/mspec/bin/mspec spec/ruby/core/array/each_spec.rb

Bootstrap tests

bootstraptest/ files use a tiny DSL:

assert_equal 'Array', %q{ [].class }
assert_normal_exit %q{ raise "boom" rescue nil }

These run with the smallest possible Ruby (sometimes miniruby) and are often the first to break when VM internals change. They're also the fastest signal — many maintainers run make btest after every edit to vm_*.c, compile.c, or gc.c.

test/ruby and test/

Files use Test::Unit::TestCase:

require 'test/unit'

class TestArray < Test::Unit::TestCase
  def test_basic_each
    a = []
    [1, 2, 3].each { |x| a << x }
    assert_equal [1, 2, 3], a
  end
end

Convention: one test_<topic>.rb per source area. test/ruby/test_compile.rb exercises the bytecode compiler; test/ruby/test_yjit.rb exercises YJIT; etc.

tool/test/runner.rb and tool/lib/test/unit.rb extend Test::Unit with multi-process parallelism and Ruby-specific reporters.

ruby/spec

spec/ruby/ is a vendored mirror of the cross-implementation ruby/spec project. Specs document observable behaviour:

describe "Array#each" do
  it "yields each element" do
    a = []
    [1, 2, 3].each { |x| a << x }
    a.should == [1, 2, 3]
  end
end

When you change observable Ruby semantics, you typically need both a test/ruby/ test (defends CRuby's specific behaviour) and a spec/ruby/ spec (documents the language's behaviour). The spec change should be PRed both here and at ruby/spec upstream.

spec/.excludes/, spec/.excludes-mmtk/, and spec/.excludes-zjit/ exclude specs that don't apply to specific configurations. If a JIT or GC variant breaks a spec deliberately, add an exclusion file with a # reason comment.

Benchmarks

benchmark/ contains microbenchmarks, used by:

• make benchmark — runs a curated subset.
• The external yjit-bench repo for end-to-end workloads (Optcarrot, Liquid, etc.).

When tuning hot paths, run the affected benchmarks before and after with benchmark-driver (make benchmark BENCH_RUBY=./ruby BENCH_FILE=benchmark/foo.yml).

CI

.github/workflows/ defines the production CI. Notable workflows:

• ubuntu.yml, macos.yml, windows.yml, mingw.yml — cross-platform builds.
• yjit-ubuntu.yml, zjit-ubuntu.yml, yjit-macos.yml — JIT-focused configurations.
• mmtk.yml — runs test-all/test-spec with the MMTk GC.
• bsdl.yml — checks BSDL license file.
• compilers.yml — tests building under multiple GCC/Clang versions.
• prism.yml — runs the suite with the Prism parser as the default.

CI is the source of truth for "what is expected to pass". A PR that turns a workflow red won't be merged.

Coverage and leak checking

• tool/test-coverage.rb and tool/run-lcov.rb produce LCOV reports for the C code.
• make exam includes make leaked-globals (checks symbol leaks in the built binary).
• make test-rubyspec is an alias for make test-spec.

Adding a new test

For a new method Array#sample_with_seed:

1. Add a Test::Unit test in test/ruby/test_array.rb covering happy paths and edge cases.
2. Add a spec under spec/ruby/core/array/sample_with_seed_spec.rb.
3. If the change is YJIT-relevant, add a regression test in test/ruby/test_yjit.rb.

For a new public C API:

1. Add a documented declaration in the relevant include/ruby/ header.
2. Add C tests under ext/-test-/ (the -test- extension is built only during make test-all).

Excluded tests

Some tests are skipped on specific platforms or configurations:

• spec/.excludes/ — generic spec exclusions.
• tool/rbs_skip_tests / tool/rbs_skip_tests_windows — skip flaky RBS tests on Windows.
• test/excludes/ (per default-gem subdirectories) — skip a particular known-failing test from a vendored gem.

When you add an exclusion, leave a comment explaining the failure mode.