Stream proxy

Active contributors: Sergey Kandaurov, Roman Arutyunyan, Vladimir Homutov

Purpose

NGINX's "stream" subsystem is a generic L4 (TCP and UDP) proxy and load balancer. Unlike the HTTP server, it doesn't parse application-layer protocols — it just establishes a session, optionally peeks at the first bytes (for SNI / ALPN routing), and pipes bytes between client and backend. Used to load-balance MySQL, Redis, gRPC-over-TCP, custom binary protocols, DNS-over-UDP, and TLS-terminated workloads.

The Stream module was ported back from the commercial NGINX+ codebase in Apr 2015. First mainline commit: 2015-04-20.

Directory layout

src/stream/
├── ngx_stream.{c,h}                 # the stream { } meta-module + bootstrap
├── ngx_stream_handler.c             # connection accept + dispatch
├── ngx_stream_core_module.c         # core directives, server { } resolution
├── ngx_stream_proxy_module.c        # the proxy itself (large; ~3,800 lines)
├── ngx_stream_pass_module.c         # internal redirect to a different listener
├── ngx_stream_return_module.c       # static response (a la HTTP `return`)
├── ngx_stream_set_module.c          # variable assignment
├── ngx_stream_map_module.c          # map for stream variables
├── ngx_stream_geo_module.c          # geo for stream variables
├── ngx_stream_geoip_module.c        # MaxMind GeoIP
├── ngx_stream_split_clients_module.c  # split_clients
├── ngx_stream_log_module.c          # access log + log_format
├── ngx_stream_realip_module.c       # PROXY-protocol client IP
├── ngx_stream_access_module.c       # allow/deny by IP
├── ngx_stream_limit_conn_module.c   # connection-count limits
├── ngx_stream_ssl_module.{c,h}      # listen-side TLS
├── ngx_stream_ssl_preread_module.c  # peek at SNI/ALPN before deciding upstream
├── ngx_stream_upstream.{c,h}        # upstream framework
├── ngx_stream_upstream_round_robin.{c,h}    # default LB
├── ngx_stream_upstream_hash_module.c        # hash-based LB
├── ngx_stream_upstream_least_conn_module.c  # least-conn LB
├── ngx_stream_upstream_random_module.c      # random LB
├── ngx_stream_upstream_zone_module.c        # shared-memory upstream state
├── ngx_stream_script.{c,h}          # variable bytecode (mirrors HTTP scripts)
├── ngx_stream_variables.{c,h}       # variable registration / lookup
└── ngx_stream_write_filter_module.c # the bottom of the output chain

Key abstractions

Type / function	Role
ngx_stream_session_t	Per-connection state — analogous to ngx_http_request_t
ngx_stream_handler_pt	Function pointer signature for stream content handlers (ACCESS, PREACCESS, CONTENT phases)
ngx_stream_phase_handler_t	Phase array entry
ngx_stream_proxy_init_upstream()	Open the backend connection
ngx_stream_proxy_process()	The byte-pipe loop
ngx_stream_ssl_preread_module	Holds reads briefly to inspect TLS ClientHello

Phase model

Stream has its own (smaller) phase engine, structurally similar to HTTP's:

typedef enum {
    NGX_STREAM_POST_ACCEPT_PHASE = 0,
    NGX_STREAM_PREACCESS_PHASE,
    NGX_STREAM_ACCESS_PHASE,
    NGX_STREAM_SSL_PHASE,
    NGX_STREAM_PREREAD_PHASE,
    NGX_STREAM_CONTENT_PHASE,
    NGX_STREAM_LOG_PHASE
} ngx_stream_phases;

Phase	Modules attached
POST_ACCEPT	realip
PREACCESS	limit_conn
ACCESS	access
SSL	ssl (terminate inbound TLS)
PREREAD	ssl_preread (peek at first bytes for routing)
CONTENT	proxy, pass, return
LOG	log

The PREREAD phase is unique to Stream: it lets a handler buffer the first preread_buffer_size bytes from the client and inspect them — typically to pull SNI from a TLS ClientHello so a routing decision can be made without having to terminate TLS.

How a session flows

sequenceDiagram
    participant C as Client
    participant L as Listener
    participant S as ngx_stream_handler
    participant Ph as Phase engine
    participant P as ngx_stream_proxy
    participant B as Backend

    C->>L: TCP connect (or first UDP packet)
    L->>S: ngx_stream_init_connection
    S->>Ph: walk phases
    Ph->>Ph: PREREAD: peek at SNI?
    Ph->>P: CONTENT: ngx_stream_proxy_handler
    P->>B: connect
    B-->>P: connected
    loop until either side closes
        C-->>P: bytes
        P-->>B: bytes
        B-->>P: bytes
        P-->>C: bytes
    end
    Ph->>Ph: LOG

The proxy module's process loop is a straightforward back-pressured copy: read available bytes from one side, write to the other, swap directions. UDP gets its own minor variant since each "session" is just a series of datagrams under one source-address tuple.

TCP and UDP

listen <addr>:<port>; is TCP; listen <addr>:<port> udp; is UDP. UDP "sessions" are tracked by source 4-tuple (or 5-tuple with proxy_responses) and time out after proxy_timeout.

DNS-over-UDP load balancing is the canonical stream-UDP use case.

SSL preread

ngx_stream_ssl_preread_module reads the first up-to-preread_buffer_size bytes, parses the TLS ClientHello, extracts the SNI hostname / ALPN list, and exposes them as variables ($ssl_preread_server_name, $ssl_preread_alpn_protocols). Combined with map, this lets you route myhost.example.com:443 to one backend pool and otherhost.example.com:443 to another without terminating TLS.

Variables and scripts

Stream has its own variable system (src/stream/ngx_stream_variables.c) and its own script bytecode engine (src/stream/ngx_stream_script.c) — structurally identical to the HTTP versions, but separated because the contexts and lifetimes differ. Variables prefixed $ssl_*, $proxy_*, $bytes_sent, $session_time come from here.

Upstream

ngx_stream_upstream.c mirrors the HTTP upstream framework but is much smaller (~700 lines). It supports the same load-balancing modules (round-robin, hash, least-conn, random, zone) and the same retry semantics (proxy_next_upstream).

Configuration shape

stream {
    upstream backend {
        server 10.0.0.1:5432 weight=2;
        server 10.0.0.2:5432;
        server 10.0.0.3:5432 backup;
    }

    server {
        listen     5432;
        proxy_pass backend;
    }

    map $ssl_preread_server_name $name {
        api.example.com  api_pool;
        web.example.com  web_pool;
        default          default_pool;
    }

    server {
        listen     443;
        ssl_preread on;
        proxy_pass $name;
    }
}

Integration points

• Event loop — same ngx_event_t machinery.
• OpenSSL — ngx_stream_ssl_module for inbound TLS termination, proxy_ssl_* directives for outbound to backends.
• Resolver — for hostname-based upstreams.
• Configuration system — stream { } is its own block under NGX_STREAM_MODULE type.

Entry points for modification

Most additions are new content handlers (e.g., a protocol-specific filter that does more than just pipe bytes) or new load balancers. The phase engine and the proxy itself are stable. The PREREAD phase is the right hook for any feature that wants to make routing decisions on the first bytes of a connection.