TLS

Active contributors: Matt Caswell, Tomas Mraz, Bernd Edlinger, Hugo Landau, Neil Horman, Stephen Henson (historical)

Purpose

libssl implements:

• TLS 1.0, 1.1, 1.2, 1.3 (RFC 8446) — client and server.
• DTLS 1.0, 1.2 — client and server.
• ECH (Encrypted Client Hello, RFC 9460 + draft) — client and server, since 3.6 / merged 2026-02.
• TLS extensions: ALPN, SNI, supported_groups, signature_algorithms, key_share, PSK, early data, post-handshake auth, certificate compression, OCSP stapling, SCT, RPK, certificate authorities, server-name-indication, …
• Hybrid post-quantum key exchange via the TLS-GROUP capability (e.g. X25519MLKEM768 since 3.5).

The implementation is split into three layers:

1. API & state in ssl/ (the SSL, SSL_CTX, SSL_METHOD, SSL_SESSION types).
2. State machine in ssl/statem/ — the explicit handshake automaton.
3. Record layer in ssl/record/ — framing, encryption, sequence numbers, replay defence (DTLS), early-data handling.

Directory layout

ssl/
├── ssl_lib.c          226 KB  -- the SSL/SSL_CTX surface, accessor zoo
├── ssl_local.h        114 KB  -- the gigantic private header
├── s3_lib.c           149 KB  -- TLS 1.0–1.2 method tables and ciphersuite table
├── ssl_ciph.c          70 KB  -- ciphersuite parsing, cipher-list configuration
├── ssl_conf.c          38 KB  -- the `Cipher`, `Protocol`, `Curves`, … config commands
├── ssl_cert.c          37 KB  -- certificate chain handling, CA list assembly
├── ssl_sess.c          41 KB  -- session caching, ticket encrypt/decrypt
├── ssl_rsa.c           30 KB  -- key/cert loading helpers (PEM/DER/PKCS#8/PKCS#12)
├── t1_lib.c           169 KB  -- TLS 1.x extension handling, group/sigalg negotiation, OCSP, SCT
├── t1_enc.c            19 KB  -- the legacy 1.0–1.2 PRF
├── tls13_enc.c         33 KB  -- HKDF labels, secret derivation, key-update
├── t1_trce.c           63 KB  -- packet trace pretty-printer (`SSL_set_msg_callback`)
├── d1_lib.c            29 KB  -- DTLS-specific bits
├── d1_msg.c, d1_srtp.c
├── methods.c           -- the SSL_METHOD vtable assembly
├── pqueue.c, priority_queue.c -- priority queues used by the DTLS retransmit logic
├── ssl_asn1.c          -- session DER encoder/decoder
├── ssl_init.c, ssl_stat.c, ssl_txt.c, ssl_utst.c
├── tls_depr.c, tls_srp.c, ssl_err_legacy.c, ssl_mcnf.c, ssl_rsa_legacy.c
├── ssl_cert_comp.c     -- certificate compression (RFC 8879)
├── bio_ssl.c           -- BIO_f_ssl filter

ssl/statem/
├── statem.c            -- the state machine driver
├── statem_clnt.c      147 KB  -- client side
├── statem_srvr.c      157 KB  -- server side
├── statem_lib.c        97 KB  -- shared helpers
├── statem_dtls.c       46 KB  -- DTLS-specific transitions and reassembly
├── extensions.c        77 KB  -- common extension dispatch
├── extensions_clnt.c   95 KB  -- client extension construction/parsing
├── extensions_srvr.c   87 KB  -- server extension construction/parsing
├── extensions_cust.c   21 KB  -- custom-extension hooks for applications
└── statem_local.h      24 KB

ssl/record/
├── rec_layer_s3.c      50 KB  -- the SSL3/TLS record layer
├── rec_layer_d1.c      22 KB  -- the DTLS record layer
├── methods/            -- per-version record method (tls_common.c, tls_pad.c, tls13_meth.c, dtls_meth.c, tls1_meth.c, …)
├── record.h, record_local.h

ssl/ech/                 -- Encrypted Client Hello (3.6+)
├── ech_internal.c      80 KB  -- core ECH machinery
├── ech_store.c         39 KB  -- ECHConfigList store / file loading
├── ech_ssl_apis.c, ech_helper.c, ech_local.h

ssl/quic/                -- QUIC v1 (see features/quic)
ssl/rio/                 -- the asynchronous reactor I/O used by the modern record layer

Public types

Type	Role	Header
SSL_CTX	Per-process configuration: certificates, ciphersuites, callbacks, session cache, default extensions.	include/openssl/ssl.h.in
SSL	Per-connection state: state machine cursor, derived keys, extension state, attached BIOs.	same
SSL_METHOD	A vtable choosing protocol version + role. TLS_method, TLS_client_method, TLS_server_method, DTLS_method, …	same
SSL_SESSION	A resumable session (master secret/ticket + selected params).	ssl_sess.c
SSL_CIPHER	An IANA ciphersuite, with name aliases and the set of EVP primitives it implies.	ssl_ciph.c
BIO_f_ssl	A filter BIO that wraps an SSL * so chained APIs can use TLS transparently.	ssl/bio_ssl.c

TLS 1.3 handshake (server-authenticated, no resumption)

sequenceDiagram
    participant C as Client (statem_clnt.c)
    participant S as Server (statem_srvr.c)
    C->>S: ClientHello {key_share=X25519, cipher_suites, sig_algs, supported_groups, ALPN, SNI}
    S->>C: ServerHello {key_share=X25519, selected_cipher}
    Note over C,S: derive handshake_secret; encrypt subsequent records
    S->>C: EncryptedExtensions {ALPN, …}
    S->>C: Certificate
    S->>C: CertificateVerify (signature over transcript)
    S->>C: Finished (HMAC over transcript)
    C->>S: [Certificate / CertificateVerify if mTLS]
    C->>S: Finished
    Note over C,S: derive application_secret; switch to app keys
    C-->>S: Application data
    S-->>C: Application data

The TLS 1.3 handshake state machine in ssl/statem/ is explicitly enumerated — the OSSL_HANDSHAKE_STATE (in ssl_local.h) lists every legal cursor (TLS_ST_CW_CLNT_HELLO, TLS_ST_CR_SRVR_HELLO, TLS_ST_OK, …). state_machine() in statem.c repeatedly:

1. Consumes one record's worth of bytes.
2. Validates the next message is allowed in the current state.
3. Calls the per-state handler in statem_clnt.c / statem_srvr.c.
4. Advances the cursor.

The state machine returns control to the caller (via SSL_read/SSL_write/SSL_do_handshake) every time it would block, and resumes from where it left off — TLS in OpenSSL is non-blocking by default at the protocol layer, regardless of whether the BIO is blocking.

Extensions

ssl/statem/extensions.c keeps a table mapping each TLS extension id to a (parse, construct, init, final, role-mask, version-mask) tuple. The dispatcher in tls_collect_extensions() walks every received extension once and calls the parser; later, tls_construct_extensions() calls the constructors in deterministic order. This central table is the place to look when adding a new extension.

The split between extensions_clnt.c and extensions_srvr.c is by who is constructing it — the client constructs the extensions in its ClientHello, the server constructs in ServerHello/EE/Cert/CertVerify, and the parsing side is the opposite.

Each extension's per-direction constructor and parser are registered in the table in extensions.c. Searching for TLSEXT_TYPE_<name> finds the entry.

Ciphersuites

ssl/s3_lib.c defines ssl3_ciphers[], a static table of every supported ciphersuite (TLS 1.2 and TLS 1.3 alike). Each row carries:

• IANA id (the wire codepoint).
• Name and alternate names (for OpenSSL cipher list parsing).
• Algorithm components (KEX, AUTH, ENC, MAC, version mask, strength bits).
• Per-protocol min/max version.

The cipher-list configuration string (HIGH:!aNULL:!eNULL:…) is parsed by ssl_ciph.c:SSL_CTX_set_cipher_list(), which produces an ordered subset of the table.

For TLS 1.3, ciphersuites are decoupled from the key exchange and signature algorithms; the latter come from the TLS-GROUP and TLS-SIGALG capability tables published by providers (providers/common/capabilities.c) and assembled in ssl/t1_lib.c.

Record layer

The record layer is responsible for:

• Framing TLS records (5-byte header + ciphertext).
• Encrypting/decrypting records (AEAD for 1.3, MAC-then-encrypt or stitched for 1.2 and below).
• Sequencing records (sequence number is per-key-epoch).
• Padding and length-hiding (TLS 1.3).
• Early-data routing.
• For DTLS: replay window, retransmission queue, MTU-aware fragmentation.

The 3.0 design split the record layer out from the rest of libssl so that it can be replaced for QUIC. The current API to the record layer is the OSSL_RECORD_METHOD (ssl/record/record.h), which has separate implementations for TLS 1.0, 1.1, 1.2, 1.3, DTLS 1.0, DTLS 1.2, and "QUIC". Each lives in ssl/record/methods/.

rec_layer_s3.c is the legacy in-tree record layer used by TLS pre-3.0 — the s3_*.c naming dates back to SSLv3.

DTLS

Datagram-TLS lives alongside TLS in the same statem files but takes special transitions handled in statem_dtls.c. Key differences:

• Each record has an explicit 8-byte sequence number for replay defence.
• Handshake messages are fragmented into records that may arrive out of order; the queue lives in pqueue.c and priority_queue.c.
• The handshake retransmit timer uses an exponential backoff.
• Cookies (HelloVerifyRequest in DTLS 1.0, HelloRetryRequest in DTLS 1.2/TLS1.3) defend against amplification attacks.

ECH (Encrypted ClientHello)

ssl/ech/ implements ECH per the latest IETF draft and RFC 9460 SVCB DNS records. The ECH config (a public key + cipher choice + "outer name") is loaded into SSL_CTX via SSL_CTX_ech_* APIs (ech_ssl_apis.c). At handshake time:

• The client's inner ClientHello (the real one) is HPKE-encrypted under the ECH public key.
• The outer ClientHello has a benign SNI.
• The server's EncryptedExtensions carries the inner ClientHello's transcript hash, completing the binding.

The implementation is large (ech_internal.c is 80 KB) because ECH touches almost every extension construction site (it has to defer encryption until after all extensions are decided) and adds its own cookie handling, retry config, and fallback. The store (ech_store.c) loads ECHConfigList files in PEM or wire format.

Session resumption

TLS 1.3 supports resumption via tickets (RFC 8446 §4.6.1) and PSK; older versions also have session-id resumption. The implementation in ssl/ssl_sess.c plus ssl/statem/statem_lib.c:

• Server issues NewSessionTicket after handshake completion (TLS 1.3) or in ServerHello (TLS 1.2).
• Tickets are encrypted with a server-private session ticket key; rotated via SSL_CTX_set_tlsext_ticket_key_evp_cb.
• Client stores the ticket+session params in SSL_SESSION and presents pre_shared_key extension on resume.

Renegotiation and key-update

TLS 1.2 supports renegotiation (and the secure renegotiation extension RFC 5746 — never disable it). TLS 1.3 replaces this with KeyUpdate (tls13_enc.c:tls13_update_keys).

Configuration model

SSL_CTX_load_verify_locations(ctx, "/etc/ssl/certs", NULL);
SSL_CTX_set_cipher_list(ctx, "ECDHE+AESGCM:ECDHE+CHACHA20:!aNULL:!RC4");
SSL_CTX_set_ciphersuites(ctx, "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256");
SSL_CTX_set_min_proto_version(ctx, TLS1_2_VERSION);
SSL_CTX_set_max_proto_version(ctx, TLS1_3_VERSION);
SSL_CTX_set1_groups_list(ctx, "X25519MLKEM768:X25519:P-256");
SSL_CTX_use_certificate_chain_file(ctx, "server-fullchain.pem");
SSL_CTX_use_PrivateKey_file(ctx, "server-key.pem", SSL_FILETYPE_PEM);

The ssl/ssl_conf.c module also exposes the same options in a string-keyed form usable from openssl.cnf.

Tracing handshakes

SSL_set_msg_callback(ssl, my_cb) plugs a callback into every received and sent record. The default tracer in ssl/t1_trce.c decodes every TLS message back into pseudo-text — it is essentially a built-in tshark for OpenSSL.

SSL_CTX_set_msg_callback(ctx, SSL_trace);
SSL_CTX_set_msg_callback_arg(ctx, BIO_new_fp(stderr, BIO_NOCLOSE));

TLS-GROUP and TLS-SIGALG

The set of supported TLS 1.3 groups (X25519, MLKEM768, X25519MLKEM768, P-256, …) and signature schemes (rsa_pss_rsae_sha256, ecdsa_secp256r1_sha256, ed25519, mldsa44, …) is published by providers, not hardcoded in libssl. Providers register a TLS-GROUP or TLS-SIGALG capability via OSSL_OP_PROVIDER capabilities. ssl/t1_lib.c queries the libctx's loaded providers at handshake time. This is how third-party providers can teach OpenSSL about new TLS 1.3 groups (oqs-provider).

Integration points

• Below: every cryptographic primitive is fetched via EVP from the configured providers. The TLS state machine never touches algorithm internals directly.
• Sidewise: the record layer is selected from ssl/record/methods/ based on protocol version. The QUIC implementation reuses the same TLS handshake state machine but with the QUIC record method (which framing is via QUIC CRYPTO frames, not TLS records).
• Above: applications use SSL_* (or BIO_f_ssl). Web servers (Apache, Nginx), language runtimes (Python ssl, Node.js TLS), embedded devices, etc.

Entry points for modification

• New TLS extension: add a TLSEXT_TYPE_* constant in include/openssl/tls1.h.in, add a row to the dispatcher table in ssl/statem/extensions.c, write tls_construct_ctos_<name> / tls_parse_ctos_<name> / tls_construct_stoc_<name> / tls_parse_stoc_<name> in extensions_clnt.c / extensions_srvr.c. Update ssl_local.h for any per-SSL state.
• New ciphersuite: add a row in ssl3_ciphers[] (s3_lib.c) and corresponding SSL_kEX*, SSL_aAUTH*, etc. flags. For TLS 1.3, also list the AEAD/MD pair.
• New TLS-GROUP: register it from a provider via OSSL_OP_PROVIDER capabilities, not in libssl.
• New protocol version (a hypothetical TLS 1.4): add a method, a new record-layer method in ssl/record/methods/, transitions in ssl/statem/statem_*.c, version-aware extension entries.
• DTLS retransmission tweaks: ssl/d1_lib.c and statem_dtls.c.

Documentation

• doc/man3/SSL_CTX_new.pod, SSL_new.pod, SSL_read.pod, SSL_write.pod, SSL_do_handshake.pod, SSL_CTX_set_cipher_list.pod, SSL_CTX_set1_groups.pod, SSL_CTX_set_min_proto_version.pod, …
• doc/man7/ossl-guide-tls-introduction.pod and the rest of the ossl-guide-tls-* series.
• ssl/statem/README.md — short note on the state machine philosophy.