OpenSSL integration

Active contributors: Sergey Kandaurov

Purpose

src/event/ngx_event_openssl.{c,h} is nginx's wrapper around OpenSSL. It handles the TLS handshake on read/write events, manages certificate loading, runs the OCSP stapling state machine, maintains session and OCSP response caches in shared memory, and exposes a connection's TLS state via the ngx_ssl_connection_t attached to each ngx_connection_t.

ngx_event_openssl.c is the second-largest file in the entire codebase at ~6,600 lines. The OpenSSL bindings have absorbed two-plus decades of "OpenSSL renamed this API again" patches and maintain a thin compatibility layer for OpenSSL 1.1.x, 3.x, 4.x, BoringSSL, and (via shims) LibreSSL.

Files

File	Role
src/event/ngx_event_openssl.{c,h}	Core: handshake, send/recv, session, ALPN, certificate management
src/event/ngx_event_openssl_cache.c	Cert / pkey / OCSP / DH / ECDH parameter caches
src/event/ngx_event_openssl_stapling.c	OCSP stapling and OCSP responder client
src/event/quic/ngx_event_quic_openssl_compat.{c,h}	Shim for OpenSSLs that lack QUIC keying APIs

Key abstractions

Type	Role
ngx_ssl_t	Per-cycle OpenSSL SSL_CTX * plus nginx state
ngx_ssl_connection_t	Per-connection state: the SSL *, the handshake handler, buffers
ngx_ssl_session_cache_t	Shared-memory rbtree for session resumption
ngx_ssl_handshake()	Drive a non-blocking TLS handshake on a connection
ngx_ssl_recv() / ngx_ssl_send()	TLS-aware replacements for the OS recv/send
ngx_ssl_certificate()	Load a PEM certificate (and chain) into the SSL_CTX
ngx_ssl_session_cache_init()	Set up the shared-zone session cache
ngx_ssl_stapling() / ngx_ssl_stapling_responder()	OCSP stapling

How a TLS handshake runs

sequenceDiagram
    participant E as event loop
    participant H as ngx_ssl_handshake
    participant SSL as OpenSSL
    participant K as Kernel

    H->>SSL: SSL_do_handshake
    SSL->>K: write ServerHello + Certificate + ...
    SSL->>K: read ClientHello
    SSL-->>H: WANT_READ
    H->>E: arm c->read
    E-->>H: read event fires
    H->>SSL: SSL_do_handshake (again)
    SSL->>K: write Finished
    SSL-->>H: SUCCESS
    H->>H: ngx_ssl_handshake_handler -> protocol init

ngx_ssl_handshake is reentrant: each call advances OpenSSL's state machine, and on SSL_ERROR_WANT_READ / WANT_WRITE the function arms the appropriate event and returns. The next event firing re-enters the handshake. When the handshake completes, the continuation handler (c->ssl->handler) gets called — for HTTP that's ngx_http_ssl_handshake_handler, which dispatches to either H1 or H2 based on negotiated ALPN.

Send/recv

After handshake, c->recv = ngx_ssl_recv, c->send = ngx_ssl_send, etc. These wrap SSL_read / SSL_write and translate SSL_ERROR_WANT_READ / WANT_WRITE into nginx's NGX_AGAIN semantics. c->send_chain is replaced with a chain-aware variant that minimizes SSL_write calls by buffering small chains together (controlled by ssl_buffer_size).

The bytes read from the socket aren't necessarily returned by SSL_read — they may be a partial TLS record. OpenSSL handles this internally; nginx just keeps calling SSL_read until it returns either bytes or WANT_READ.

Certificates and SNI

ngx_ssl_certificate(cf, ssl, &cert, &key, NULL) loads a certificate. The function's behavior depends on the path argument: a literal file gets loaded once at startup; a path with variables (a "complex value") gets loaded per request via the SSL_CTX_set_cert_cb SNI callback. This is how ssl_certificate $ssl_server_name.pem; works.

ngx_ssl_servername() is the SNI callback. On each ClientHello, it looks up $ssl_server_name in the per-server config tree and switches the SSL_CTX accordingly.

Session caches

Three flavors of TLS session state, each backed by shared memory:

1. Session ID cache — the classical TLS 1.2 mechanism. ngx_ssl_session_cache_init allocates a slab pool, builds a rbtree keyed by session ID. Hits return the resumed master secret; expired entries get evicted by the manager process.
2. Session tickets — TLS 1.2/1.3 tickets are encrypted server-side. Keys are configured via ssl_session_ticket_key (rotate them), implemented in ngx_ssl_session_ticket_keys.
3. TLS 1.3 PSK resumption — built on the ticket machinery; same key handling.

OCSP stapling

ngx_event_openssl_stapling.c (~2,800 lines) is the largest single file in src/event/ outside QUIC. It:

1. On config load, optionally fetches an OCSP response for each cert (ssl_stapling_responder URL).
2. Caches the response in shared memory.
3. Refreshes before expiry.
4. Attaches the response to the TLS handshake via SSL_set_tlsext_status_type/SSL_set_tlsext_status_ocsp_resp.

The OCSP responder client is itself a small HTTP/1.0 client built on the resolver and connection helpers.

OpenSSL version compatibility

The codebase carries #if OPENSSL_VERSION_NUMBER shims for:

• 1.0.x → 1.1.x — opaque structs, SSL_CTX_set_* renames
• 1.1.x → 3.x — provider-based crypto, deprecated SSLv3 APIs
• 3.x → 4.x — error code renames (recent commits like abc72c5a5 SSL: compatibility with renamed error codes in OpenSSL 4.0)
• BoringSSL — different ALPN/SNI APIs in places, no OCSP stapling client

The auto/lib/openssl/conf script detects the version at build time. Most of the version-specific code is small #if blocks; OCSP, session ticket keys, and the QUIC keying-material APIs are the largest.

QUIC integration

When QUIC is enabled, the same ngx_ssl_t is used to drive TLS 1.3 inside the QUIC packet protection layer. OpenSSL's SSL_provide_quic_data / SSL_process_quic_post_handshake (when available) are used directly; if the linked OpenSSL doesn't expose them, ngx_event_quic_openssl_compat.c provides a userspace shim that intercepts the handshake bytes via BIOs and feeds them to QUIC's keying APIs. The compat shim is one of the more involved pieces of the codebase — necessary for letting upstream OpenSSL (which historically didn't expose QUIC-specific APIs) work for HTTP/3.

Configuration shape (HTTP / TLS example)

http {
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;             # or on with ssl_session_ticket_key

    server {
        listen              443 ssl;
        server_name         example.com;
        ssl_certificate     /etc/nginx/cert.pem;
        ssl_certificate_key /etc/nginx/key.pem;

        ssl_stapling          on;
        ssl_stapling_verify   on;
    }
}

Integration points

• Event loop — handshake is just an event handler. Send/recv replace c->recv / c->send once the handshake completes.
• HTTP / Mail / Stream — all three subsystems have their own ssl_module that just wires this layer in.
• QUIC — uses the same ngx_ssl_t and the OpenSSL compat shim.
• Configuration — ssl_* directives are scattered across HTTP, Mail, Stream modules; the heavy lifting is in this file.

Entry points for modification

OpenSSL version bumps mostly land in auto/lib/openssl/ (probes) and ngx_event_openssl.c (#if branches). Adding a new TLS feature (e.g., a new ALPN protocol that doesn't fit h2/http/1.1/h3) goes through the protocol-specific module's init callback. Don't try to add new OpenSSL features in auto/lib/openssl/; that's for probing what's already in OpenSSL, not for shipping crypto.