Debugging

How to find out what OpenSSL is doing when it does not behave as you expect.

Step 1: Read the error stack

Whenever an OpenSSL function returns failure (typically 0 or NULL), the first thing to do is dump the error stack:

ERR_print_errors_fp(stderr);

Or in C++:

unsigned long e;
char buf[256];
while ((e = ERR_get_error()) != 0) {
    ERR_error_string_n(e, buf, sizeof(buf));
    fprintf(stderr, "openssl: %s\n", buf);
}

Each error has a library, a reason, and (when compiled with enable-err-stack-trace) a file/line. The reason text comes from per-library *err.h / *_err.c files; new contributors will edit these when they add error codes (see development-workflow).

The openssl errstr <hex> subcommand decodes a single error code from a log, and apps/errstr.c is its implementation.

Step 2: Turn on the trace facility

Compile with enable-trace:

./Configure enable-trace --debug
make -j$(nproc)

There are a few dozen named trace categories (declared in include/openssl/trace.h):

TRACE BN, CMP, CONF, ENGINE_CONF, ENGINE_REF, ENGINE_TABLE, INIT,
DECODER, ENCODER, REF_COUNT, PKCS5V2, PKCS12_KEYGEN, PKCS12_DECRYPT,
X509V3_POLICY, BN_CTX, PROVIDER, STORE, STORE_LOADER, HTTP, TLS,
TLS_CIPHER, QUERY, OSSL_DECODER, OSSL_ENCODER, …

To listen on one or more of them:

#include <openssl/trace.h>

OSSL_trace_set_channel(OSSL_TRACE_CATEGORY_TLS, BIO_new_fp(stderr, BIO_NOCLOSE));
OSSL_trace_set_channel(OSSL_TRACE_CATEGORY_PROVIDER, BIO_new_fp(stderr, BIO_NOCLOSE));

crypto/trace.c is the implementation. Each category corresponds to OSSL_TRACE_BEGIN(category) / OSSL_TRACE_END blocks scattered through libcrypto.

Step 3: TLS-specific tracing

Two layers of TLS visibility:

1. -trace / enable-ssl-trace. Compile-time flag; emits a human-readable transcript of every TLS message. Implementation in ssl/t1_trce.c (~65 KB).
2. -msg in s_client/s_server. Always-available; less detail but no recompilation needed.

openssl s_client -connect example.com:443 -msg -trace

For a programmatic equivalent, register a message callback:

SSL_CTX_set_msg_callback(ctx, my_cb);
SSL_CTX_set_msg_callback_arg(ctx, stderr);
SSL_CTX_set_msg_callback(ctx, SSL_trace);   /* uses ssl/t1_trce.c */

For QUIC, use qlog:

SSL_QLOG_DIR=/tmp/qlog openssl s_client -quic -alpn h3 -connect host:443

The qlog writer is in ssl/quic/qlog.c. The schema follows the IETF QUIC qlog draft; tools like qvis can render the resulting JSON.

Step 4: Sanitizers and valgrind

What	How
AddressSanitizer	./Configure enable-asan
UndefinedBehaviorSanitizer	./Configure enable-ubsan
MemorySanitizer	./Configure enable-msan
Valgrind	make OSSL_USE_VALGRIND=yes test, plus the suppression file util/valgrind.suppression

For valgrind specifics — including how OpenSSL deliberately introduces "uninitialized" bytes for entropy and how to silence those — see NOTES-VALGRIND.md.

Step 5: GDB and the test harness

Each test program is a normal binary. Run it directly under gdb (the wrapper sets LD_LIBRARY_PATH for you):

gdb --args ./util/wrap.pl test/sslapitest -test foo_subtest

For a Perl recipe, find the underlying invocation by re-running with V=1:

make TESTS=test_x509 V=1 test 2>&1 | tee log
grep -E "(util/wrap.pl|openssl)" log

The util/wrap.pl invocations there can be copied into a gdb command line.

Step 6: When the answer is in make update

A class of confusing failures comes from forgetting to run make update after adding or renaming public symbols. Symptoms:

• Linker errors about an OSSL_FUNC_* ID that is not in libcrypto.num.
• CI's update-check job complaining that generated files are stale.

Cure:

make update
git diff util/libcrypto.num util/libssl.num crypto/err/openssl.txt

Commit the generated changes alongside your code change.

Common gotchas

Symptom	Likely cause
"no algorithms available" / unable to load XXX	No provider loaded with the algorithm. Load default (or legacy for old algorithms).
TLS 1.3 server replies but client gets EOF	TLS 1.3 sends NewSessionTicket post-handshake. If your application closes the connection before reading it, sessions can't resume.
"FIPS module integrity check failed"	fipsmodule.cnf is stale (the fips.so was rebuilt). Re-run openssl fipsinstall. See providers/fips.
QUIC loops in poll	Make sure your application handles SSL_get_event_timeout() and SSL_handle_events(). See doc/man7/openssl-quic.pod.
Code compiles but EVP_CIPHER_fetch returns NULL for an algorithm you can see in openssl list -cipher-algorithms	Property query mismatch. Check EVP_default_properties_* and OSSL_LIB_CTX.
ABI mismatch between two OpenSSL builds	Different --api= levels; different enable-deprecated posture; different OPENSSL_NO_* flags. Inspect include/openssl/opensslconf.h from both builds.