Debugging

How to figure out why nginx is doing what it's doing.

The debug log

This is the single most useful tool. It dumps a full per-request trace of every event-loop wakeup, parsing decision, upstream interaction, and filter chain step.

To get it:

1. Build with --with-debug. (nginx -V will tell you whether you have it.)
2. Set error_log /path/to/error.log debug; (or error_log stderr debug;).
3. Trigger one request — keep the volume small, this is loud.

You can scope debug to one client IP:

events {
    debug_connection 192.168.1.1;
}
error_log logs/error.log;        # default level

Then only connections from 192.168.1.1 are logged at debug while everything else stays at the global level. Useful in production.

The debug categories are baked in via macros: ngx_log_debug0, ngx_log_debug1, ..., with NGX_LOG_DEBUG_HTTP, NGX_LOG_DEBUG_EVENT, NGX_LOG_DEBUG_CORE, etc. Each subsystem prefixes its messages so grep "http upstream" or grep "epoll" works well.

gdb on the running binary

master_process off; plus daemon off; gives you a single foreground process you can attach to:

daemon off;
master_process off;
worker_processes 1;

gdb --args ./objs/nginx -p $PWD -c conf/nginx.conf
(gdb) run

For attaching to a worker spawned by the master:

ps -o pid,ppid,cmd -ax | grep nginx
gdb -p <worker_pid>

A few well-named breakpoints when starting cold:

Breakpoint	When you'd set it
ngx_http_init_request	Each new HTTP request
ngx_http_finalize_request	When a request ends (look at rc)
ngx_http_upstream_init	Beginning of an upstream subrequest
ngx_http_upstream_finalize_request	End of an upstream subrequest
ngx_event_accept	Before each accepted connection
ngx_ssl_handshake	TLS handshake on inbound
ngx_ssl_handshake_handler	TLS handshake completion
ngx_quic_run	QUIC connection start

Useful runtime flags

Flag	Effect
nginx -t	Test config and exit
nginx -T	Dump fully-included config
nginx -V	Print version + configure args
daemon off;	Run master in foreground
master_process off;	Skip the master entirely (single process for debug)
worker_processes 1;	One worker — gdb attach is unambiguous
error_log stderr debug;	Send full debug trace to terminal
worker_connections 1024;	Lower if you're hitting fd ceilings under tooling

Common error log messages, decoded

Message	Meaning
worker_connections is not enough	More concurrent connections than the worker pool can hold; raise it
accept() failed (24: Too many open files)	Hit OS fd limit; raise worker_rlimit_nofile
upstream prematurely closed connection	Backend hung up mid-response; usually a backend issue, not nginx
client intended to send too large body	Bigger than client_max_body_size
[alert] ... sendmsg() failed (90: Message too long)	Channel message between master/worker exceeded a buffer (rare)
SSL_do_handshake() failed (SSL: error:0A0000B6:SSL ...)	TLS handshake error from OpenSSL — code identifies the specific fault

strace / dtrace / perf

For performance work the debug log is too slow to leave on in production. Common alternatives:

• strace -p <worker_pid> -f -e trace=network,read,write — quick view of syscalls
• perf record -p <worker_pid> -F 99 -g -- sleep 30 && perf report — flamegraph fodder
• DTrace probes on FreeBSD / macOS — no built-in USDT in nginx, but the kqueue modules emit standard kevent traces

Memory leaks and corruption

• Build with -fsanitize=address,undefined (see testing).
• Run a representative workload under the sanitizer build.
• Errors point to a stack trace; map back to the source line.
• For pool-allocated memory leaks: every leak is technically a feature (pools free everything when destroyed), so the question is "did we destroy the pool we should have?" rather than "did we free this allocation?". Connection pools, request pools, and subrequest pools are the usual suspects. Look for code paths that bypass ngx_http_finalize_request or ngx_close_connection.

When in doubt: read the source

Three tactical reads that pay off repeatedly:

• src/http/ngx_http_request.c — the request state machine
• src/http/ngx_http_upstream.c — the upstream subrequest state machine
• src/event/ngx_event.c — what the event loop does on each iteration

The code is dense but small per file (most things are 200–500 line functions) and the naming is consistent enough that grep gets you to the right place quickly.