Event loop

Active contributors: Sergey Kandaurov, Maxim Dounin, Roman Arutyunyan

Purpose

Each nginx worker runs a single event loop on a single OS thread. The loop polls for I/O readiness, fires timers, dispatches ngx_event_t handlers, and processes queues of posted events. The actual readiness mechanism is pluggable — there's an event module per OS facility (epoll, kqueue, /dev/poll, eventport, poll, select, IOCP).

Directory layout

src/event/
├── ngx_event.{c,h}             # the loop itself + ngx_event_module
├── ngx_event_accept.c          # accept() handler
├── ngx_event_acceptex.c        # Windows AcceptEx variant
├── ngx_event_connect.{c,h}     # outbound connect()
├── ngx_event_connectex.c       # Windows ConnectEx variant
├── ngx_event_pipe.{c,h}        # back-pressured pipe between client + upstream
├── ngx_event_posted.{c,h}      # ngx_post_event + posted-events queues
├── ngx_event_timer.{c,h}       # the timer rbtree
├── ngx_event_udp.{c,h}         # UDP datagram dispatch
├── ngx_event_openssl.{c,h}     # OpenSSL bindings (see openssl.md)
├── ngx_event_openssl_cache.c   # cert / OCSP / session caches
├── ngx_event_openssl_stapling.c # OCSP stapling
├── modules/
│   ├── ngx_epoll_module.c      # Linux
│   ├── ngx_kqueue_module.c     # FreeBSD, macOS
│   ├── ngx_eventport_module.c  # Solaris
│   ├── ngx_devpoll_module.c    # Solaris (legacy)
│   ├── ngx_poll_module.c       # portable
│   ├── ngx_select_module.c     # portable fallback
│   ├── ngx_iocp_module.{c,h}   # Windows
│   ├── ngx_win32_poll_module.c # Windows poll
│   └── ngx_win32_select_module.c
└── quic/                       # see quic.md

Key abstractions

Type	File	Role
ngx_event_t	src/event/ngx_event.h	A single read or write event. Has a handler, timer node, ready/active flags
ngx_event_actions_t	src/event/ngx_event.h	Function table the readiness backend fills in (add, del, process_events)
ngx_event_conf_t	src/event/ngx_event.h	The events { } block: worker_connections, accept_mutex, multi_accept
ngx_process_events_and_timers()	src/event/ngx_event.c	The body of one event-loop iteration
ngx_post_event() / ngx_event_process_posted()	src/event/ngx_event_posted.h	Defer an event handler to run after the current one returns
ngx_event_add_timer() / ngx_event_expire_timers()	src/event/ngx_event_timer.h	Timer scheduling (rbtree-backed)
ngx_handle_read_event() / ngx_handle_write_event()	src/event/ngx_event.h	Re-arm an event (handles edge-vs-level differences)

How one iteration works

sequenceDiagram
    participant L as ngx_process_events_and_timers
    participant T as Timer rbtree
    participant K as Kernel (epoll/kqueue)
    participant H as Event handlers
    participant P as Posted-event queue

    L->>T: find next deadline
    L->>K: process_events(timer = next deadline)
    K-->>L: list of ready events
    L->>L: ngx_time_update()
    L->>H: for each ready ev: ev->handler(ev)
    L->>P: drain ngx_posted_accept_events
    L->>P: drain ngx_posted_events
    L->>T: expire timers <= now
    L->>L: loop again

The body lives in ngx_process_events_and_timers() (src/event/ngx_event.c):

1. Compute the timeout to next timer.
2. Optionally take the accept mutex if accept_mutex on; (legacy load balancing across workers; on modern Linux with EPOLLEXCLUSIVE or SO_REUSEPORT it's unnecessary).
3. Call the backend's process_events() (e.g., ngx_epoll_process_events). Ready events are either dispatched immediately or posted to a queue.
4. Update ngx_current_msec from gettimeofday().
5. Run ngx_event_expire_timers() for any timer that fired during the wait.
6. Drain ngx_posted_accept_events (deferred accept handlers) and ngx_posted_events (everything else).

The split into "posted" vs "immediately dispatched" lets a backend decide stack-depth strategy. epoll and kqueue post most events; select and poll dispatch in place.

Connections

Each worker pre-allocates worker_connections instances of ngx_connection_t plus parallel arrays read_events[i], write_events[i]. The free list is cycle->free_connections (see src/core/ngx_cycle.h). On accept():

1. Pop a connection from the free list.
2. Bind c->fd, c->log, c->pool, c->read = &read_events[i], c->write = &write_events[i].
3. Call the listener's per-protocol handler (ngx_http_init_connection, ngx_mail_init_connection, ngx_stream_init_connection).

When the connection ends, ngx_close_connection (src/core/ngx_connection.c) cancels timers, removes the events from the kernel, destroys the pool, and pushes the slot back onto the free list.

ngx_accept_disabled = ngx_cycle->connection_n / 8 - ngx_cycle->free_connection_n — when a worker's free list is depleted, this counter goes positive and the worker stops trying to grab the accept mutex until other connections close. This is nginx's load-shedding mechanism.

Backends

Every backend implements ngx_event_actions_t:

Module	OS	Notes
ngx_epoll_module	Linux	Default. Edge-triggered (EPOLLET). Optional EPOLLEXCLUSIVE for thundering-herd-free accept. AIO support via eventfd.
ngx_kqueue_module	FreeBSD, macOS	kqueue() with EVFILT_READ/EVFILT_WRITE. Per-event kq_errno/available populated by kernel.
ngx_eventport_module	Solaris	Solaris 10+ event ports.
ngx_devpoll_module	Solaris (legacy)	/dev/poll. Pre-eventport.
ngx_poll_module	portable	POSIX poll(). Fallback when nothing better is available.
ngx_select_module	portable	POSIX select(). Last-resort fallback.
ngx_iocp_module	Windows	I/O completion ports. Used with the AcceptEx/ConnectEx variants.
ngx_win32_poll_module	Windows	WSAPoll().
ngx_win32_select_module	Windows	select() on Windows.

Choice is driven by auto/configure plus the events { use ...; } directive. Most Linux setups end up on epoll automatically.

Posted events

ngx_post_event(ev, &ngx_posted_events) schedules ev->handler(ev) to run at the bottom of the current loop iteration rather than directly from inside the event-readiness handler. This avoids deep recursion, lets the backend hand back control, and is essential when a handler might modify the event set (e.g., closing a connection while iterating over ready events).

There are three queues:

• ngx_posted_accept_events — accepts that the listener got but hasn't dispatched yet.
• ngx_posted_events — generic posted events.
• ngx_posted_next_events — events that should run on the next iteration. Used by, e.g., the QUIC ack scheduler.

Timers

ngx_event_timer_rbtree (src/event/ngx_event_timer.c) is a red-black tree keyed by event->timer.key (= deadline in milliseconds since some epoch). ngx_add_timer inserts; ngx_event_expire_timers walks the tree from the leftmost node and fires every event whose key is ≤ ngx_current_msec. Cancellation is ngx_del_timer (just rbtree_delete).

This is an O(log n) data structure and the only one nginx uses for timeouts. There's no separate "fast timeout for a few ms" wheel — the rbtree handles everything from 1 ms to hours.

Accept mutex (legacy)

Before kernel features like SO_REUSEPORT and EPOLLEXCLUSIVE, multiple workers all watching the same listen socket would suffer thundering herd: every worker wakes up on every connection, only one wins the accept(). The accept mutex is a ngx_shmtx_t in shared memory; only the worker that holds it adds the listen sockets to its event set. Workers rotate ownership through ngx_trylock_accept_mutex.

On modern Linux, accept_mutex off; is the better default; EPOLLEXCLUSIVE or reuseport on the listen directive achieves the same load distribution without a userspace lock.

Integration points

• Connection acceptance: ngx_event_accept() is the per-listener accept handler. It calls each protocol's init_connection to hand the new socket off.
• Per-connection read/write: protocol modules attach handlers to c->read->handler and c->write->handler.
• TLS handshake uses ngx_ssl_handshake() (src/event/ngx_event_openssl.c), which itself sets up read/write events.
• QUIC is its own UDP-driven event subsystem layered on top — see systems/quic.
• Thread pool (src/core/ngx_thread_pool.c) integrates by waking the loop via ngx_notify (which is itself an ngx_event_actions_t member).

Entry points for modification

Adding a new event backend is a 1-2 day project: implement ngx_event_actions_t, register an ngx_event_module_t for auto/configure to detect, follow the structure of ngx_epoll_module.c. More common changes are tweaks to the loop body in ngx_process_events_and_timers() or to ngx_handle_read_event() / ngx_handle_write_event() to support a new readiness flag. Touch these carefully — the event loop is in every code path.