Pitfalls

A field guide to OpenSSL gotchas that have cost contributors and integrators time.

Don't compare ASN1_TIME with strcmp

ASN1_TIME is internally a string in either UTCTime (YYMMDDHHMMSSZ) or GeneralizedTime (YYYYMMDDHHMMSS[.fff]Z) format. A naive string comparison gives wrong answers around the year-2050 cutover. Use ASN1_TIME_compare(a, b) (or X509_cmp_time / X509_cmp_current_time for the most common cert-validity comparisons).

EVP_PKEY_CTX flags don't survive _init

EVP_PKEY_CTX_new_from_pkey(libctx, key, NULL);
EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PSS_PADDING);
EVP_PKEY_sign_init(ctx);          /* <-- this *resets* PSS padding */

*_init resets per-operation parameters. Set parameters after init.

RAND_bytes on legacy macros

RAND_bytes used to operate on a global RNG. In 3.0 it operates on the calling libctx's primary DRBG. If you mix non-default libctxs, use RAND_bytes_ex(libctx, ...) to be explicit.

SSL_pending is not read_pending

SSL_pending(ssl) returns the count of plaintext bytes already decrypted and buffered. It does not tell you whether more bytes are available on the underlying BIO. That check is SSL_has_pending. If you do level-triggered I/O on the underlying socket and only SSL_pending, you can deadlock when there is queued application data inside the SSL object.

Don't reuse an SSL_CTX certificate after SSL_CTX_use_certificate_file

The old API stores a single primary cert plus a chain. To install multiple identities (RSA + ECDSA), call SSL_CTX_use_certificate for each — the SSL_CTX maintains a per-pubkey-type table and selects per-handshake based on negotiated signature_algorithms. Calling _use_certificate for a key type that already exists replaces the prior entry.

X509_check_host empty result is success

int r = X509_check_host(cert, "example.com", strlen("example.com"), 0, NULL);
/* r == 1 -> match; r == 0 -> no match; r < 0 -> error. */

Many wrappers misinterpret r == 0 as success because 0 is truthy in some shells / loose-typed bindings. Always treat ≤ 0 as failure when calling from an FFI binding.

Don't free what you didn't allocate

Several OpenSSL APIs return internal pointers that you must not free. Examples:

• X509_get_subject_name(cert) — internal pointer; lifetime tied to cert.
• X509_get0_notAfter, X509_get0_pubkey, X509_get0_signature — note the _get0_ (no transfer of ownership). The _get1_ form does transfer ownership.
• EVP_CIPHER_CTX_get_cipher — internal pointer.

Conversely, _get1_* (or any *_dup / *_new_*) gives you a new reference / object that you own.

BIO_free vs BIO_free_all

BIO_free(b) frees only b, leaving anything chained behind it leaked. BIO_free_all(b) walks the chain.

Long-lived cipher contexts and IVs

EVP_EncryptInit_ex accepts a NULL for the IV when you want to reuse the same IV. Never reuse an IV across different plaintexts with the same key for AES-GCM, ChaCha20-Poly1305, or AES-CCM — you lose authenticity guarantees. Always draw a fresh IV (RAND_bytes(iv, len)) before each encryption.

For AES-GCM specifically the IV must be unique within the lifetime of the key; 96-bit random IVs are safe up to ~$2^{32}$ encryptions.

Forgetting to push errors

Inside the library, if a function fails and didn't push at least one error onto the stack, callers will see a mysterious failure with an empty stack. The convention: every return 0; / return NULL; path raises something. New code should call ERR_raise(ERR_LIB_<area>, <reason>) near the failing branch. The make errors make target catches some omissions.

ASN1_STRING_set does not validate

You can put arbitrary bytes into an ASN1_PRINTABLESTRING's value. The encoder then writes invalid DER. Use ASN1_mbstring_copy or the higher-level helpers that go through the ASN1_STRING_TABLE. The verifier can detect this at decode time but only if the string is round-tripped.

SSL_set_verify modes are bitwise

SSL_VERIFY_PEER alone makes the server request but accept missing client certs. To require a client cert, use SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT. SSL_VERIFY_CLIENT_ONCE controls behaviour on session resumption.

EVP_DigestSignFinal two-call pattern

The two-call pattern (call once with NULL to get the size, allocate, call again to fill) is consistent across OpenSSL but easy to get wrong:

size_t siglen;
EVP_DigestSignFinal(ctx, NULL, &siglen);
unsigned char *sig = OPENSSL_malloc(siglen);
EVP_DigestSignFinal(ctx, sig, &siglen);

If you skip the first call, siglen is uninitialised on input and EVP_DigestSignFinal interprets the garbage as a buffer size.

CMS streaming consumes its content BIO

CMS_ContentInfo *cms = CMS_sign(signer, key, certs, content_bio, CMS_STREAM | CMS_DETACHED);
SMIME_write_CMS(out_bio, cms, content_bio, CMS_STREAM | CMS_DETACHED);

SMIME_write_CMS with CMS_STREAM reads the content out of content_bio. After this call, content_bio is at EOF — you can't read it again.

Don't run openssl req against a .cnf you didn't write

openssl req -config x.cnf -new reads the cnf; the cnf can include prompt_* and string_mask knobs that change how the user is prompted. It can also load engines / providers via [openssl_init] / [providers]. Treat .cnf files like script files.

Validating a chain with extra "untrusted" certs requires the right call

X509_STORE_CTX_init(ctx, store, leaf, untrusted_chain) — untrusted_chain are intermediate candidates the verifier may use if needed. Don't put the leaf in there too. Don't put trust anchors in there — they belong in the store.

SSL_read returning 0 is not always end-of-stream

SSL_read returning 0 means the peer sent close_notify. To distinguish that from a transport error, call SSL_get_error:

int n = SSL_read(s, buf, sizeof(buf));
if (n <= 0) {
    int e = SSL_get_error(s, n);
    if (e == SSL_ERROR_ZERO_RETURN) /* clean shutdown */;
    else if (e == SSL_ERROR_WANT_READ || e == SSL_ERROR_WANT_WRITE) /* retry */;
    else /* error */;
}

OPENSSL_free is not free

Bytes returned from OpenSSL APIs (OPENSSL_strdup, BIO_get_mem_data after BIO_set_close(BIO_NOCLOSE), i2d_* allocations) must be freed with OPENSSL_free (and OPENSSL_clear_free for sensitive bytes). Calling free() on them works on most allocators but is undefined behaviour and breaks under the secure heap or a custom CRYPTO_set_mem_functions allocator.

STACK_OF(T) ownership

sk_X509_pop_free(stk, X509_free) frees the elements; sk_X509_free(stk) frees only the stack. Forgetting which is which leaks elements or double-frees them. There's a recurring class of CVEs across older OpenSSL versions caused by mixed conventions.

Threading: SSL * is not safe across threads

The SSL_CTX is mostly safe; an individual SSL is not. If your design wants to drive reads on one thread and writes on another, the safe approach is one-thread-per-connection or external synchronisation. (QUIC streams have looser rules — see features/quic.)

Compatibility with old OpenSSL versions

Code that should run against 1.1.1 and 3.x can use the OPENSSL_VERSION_NUMBER macro + #if OPENSSL_API_COMPAT < 0x30000000L, but be careful: the same source may need EVP_PKEY_get_BN_param on 3.x and EVP_PKEY_get_bn_param (a typo-fixed alias) on a transitional version. The migration_guide.pod lists every renamed symbol.