Testing

OpenSSL's test suite is large and parallelizable. The canonical reference is test/README.md and the developer-facing test/README-dev.md.

Running tests

make test                       # everything
make test V=1                   # full verbosity
make test VF=1                  # verbose only on failure
make test VFP=1                 # verbose on failure + progress on success
make TESTS='test_ssl*' test     # name-filtered
make TESTS='40' test            # group 40 (X.509 / CMS / …)
make TESTS='-99' test           # everything except slow group
make HARNESS_JOBS=4 test        # parallel
make list-tests                 # show what's available
make OSSL_USE_VALGRIND=yes test # under valgrind, see NOTES-VALGRIND.md

To rerun a flaky test deterministically, capture the seed reported on failure:

make OPENSSL_TEST_RAND_SEED=42 test
make OPENSSL_TEST_RAND_ORDER=42 test    # also re-randomizes test order

Test layout

test/
├── README.md                # how to run
├── README-dev.md            # how to write
├── README-external.md       # external test integrations
├── README.ssltest.md        # the legacy ssltest harness
├── *_test.c                 # ~360 standalone test programs
├── helpers/                 # test helpers (ssltestlib, etc.)
├── recipes/                 # ~330 Perl harnesses (.t files)
│   ├── 00-prep_*.t          # group 00: framework prep
│   ├── 01-test_abort.t      # group 01: trivial sanity
│   ├── …
│   ├── 70-test_quic_*.t     # group 70: QUIC
│   ├── 80-test_cmp_*.t      # group 80: CMP
│   ├── 90-test_*.t          # group 90: slow
│   ├── 95-test_external_*.t # group 95: external submodules
│   └── 99-test_*.t          # group 99: the slow group
├── ssl-tests/               # ssl_test data files (.cnf)
├── certs/                   # test certificates
├── ocsp-tests/, cms-msg/, ct/, d2i-tests/  # corpora
├── radix/                   # QUIC radix harness
├── shibboleth.pfx, *.pem    # known-bad inputs for fuzz/regression
└── ...                      # many test fixtures and inputs

Three styles of tests

1. Standalone C tests (test/*_test.c)

These use the Test::Harness-friendly framework declared in test/testutil.h. The pattern:

#include "testutil.h"

static int test_something(void)
{
    int ret = 0;
    EVP_MD_CTX *ctx = EVP_MD_CTX_new();
    if (!TEST_ptr(ctx))
        goto end;
    /* ... assertions using TEST_int_eq, TEST_mem_eq, etc. ... */
    ret = 1;
end:
    EVP_MD_CTX_free(ctx);
    return ret;
}

int setup_tests(void)
{
    ADD_TEST(test_something);
    return 1;
}

Useful macros (test/testutil.h):

Macro	Purpose
TEST_int_eq, TEST_int_ne, TEST_uint_eq, …	Numeric equality with logged context
TEST_ptr, TEST_ptr_null, TEST_ptr_eq	Pointer assertions
TEST_str_eq, TEST_mem_eq	String / byte buffer equality
TEST_true, TEST_false	Boolean assertions
ADD_TEST(name)	Register a non-parameterized test
ADD_ALL_TESTS(fn, n)	Register fn(0..n-1) as separate tests

To run a single C test directly without going through the harness:

./util/wrap.pl test/sslapitest -test foo_subtest -v

2. Perl recipes (test/recipes/*.t)

Recipes orchestrate higher-level tests. They typically run an openssl subcommand with crafted arguments and check the output, or they invoke a C test binary with data files. Idiom:

use OpenSSL::Test;
setup("test_widget");
plan tests => 3;
ok(run(test(["widget_test"])), "widget unit");
ok(run(app(["openssl", "req", "-x509", "-key", srctop_file("test", "key.pem")])),
   "openssl req --x509");

The Perl helpers OpenSSL::Test, OpenSSL::Test::Utils (under test/run_tests.pl and test/testlib/) provide run, app, test, srctop_file, data_file, bldtop_dir, etc.

3. SSL test framework (test/ssl-tests/, test/ssl_test.c)

Declarative TLS scenarios in .cnf files, driven by test/ssl_test.c reading via the templating in test/generate_ssl_tests.pl. This is the legacy harness; new TLS tests increasingly go in test/sslapitest.c directly.

Memory-allocation-failure tests (mfail)

Some tests use the ADD_MFAIL_TEST framework that exhaustively re-runs a test with one allocation made to fail at each iteration. Controlled via:

OPENSSL_TEST_MFAIL_DISABLE=1   # disable the custom allocator
OPENSSL_TEST_MFAIL_SKIP_ALL=1  # skip all mfail tests
OPENSSL_TEST_MFAIL_SKIP_SLOW=1 # skip the slow ones
OPENSSL_TEST_MFAIL_POINT=N     # only fail at point N
OPENSSL_TEST_MFAIL_START=N     # start from point N

See test/README.md and the _mfail.c files in test/.

External test suites

test/recipes/95-test_external_*.t invokes third-party suites under external/, wycheproof/, tlsfuzzer/, tlslite-ng/, cloudflare-quiche/, oqs-provider/, pkcs11-provider/, pyca-cryptography/, python-ecdsa/, gost-engine/, krb5/. These submodules are not cloned by default; run:

git submodule update --init --depth 1 -- gost-engine
make test TESTS=test_external_gost_engine

See test/README-external.md.

Fuzzing

Fuzz harnesses live in fuzz/. They share a corpus checked into fuzz/corpora/. Each harness builds against libFuzzer (with enable-fuzz-libfuzzer) or AFL (enable-fuzz-afl).

./Configure enable-fuzz-libfuzzer enable-asan
make
fuzz/asn1                      # run the ASN.1 fuzzer

The OSS-Fuzz integration is wired up in .github/workflows/oss-fuzz.yml and runs the same harnesses on Google's infrastructure.

CI matrix

The .github/workflows/ci.yml workflow runs about a dozen jobs per PR. The slower workflows (os-zoo.yml, compiler-zoo.yml, cross-compiles.yml, provider-compatibility.yml, valgrind-daily.yml, run-checker-daily.yml) run on a schedule. A PR is expected to be green on ci.yml before it can be merged.

Tips

• The harness writes a separate log per test under test-runs/. When something fails in CI, that's the first thing to inspect.
• make TESTS='test_X' V=1 test 2>&1 | tee log is the easiest way to capture a failure for triage.
• For a long-running TLS investigation, enable-trace (compile-time) gives you OPENSSL_TRACE output you can hook with OSSL_trace_set_callback. See debugging.