Mail proxy

Active contributors: Maxim Dounin, Sergey Kandaurov

Purpose

NGINX is a proxy for POP3, IMAP, and SMTP clients. It accepts a client connection, authenticates the user against an external HTTP "auth" endpoint, and then transparently proxies the rest of the protocol stream to a chosen backend mail server. STARTTLS and direct TLS are supported. There is no mail spool, no mailbox management — nginx is purely a front-end.

The mail subsystem landed in Mar 2007 and has been stable for years. Active changes are rare.

Directory layout

src/mail/
├── ngx_mail.{c,h}                  # the mail { } meta-module + bootstrap
├── ngx_mail_handler.c              # connection accept, IO state machine
├── ngx_mail_core_module.c          # core mail directives, server { } resolution
├── ngx_mail_parse.c                # SMTP / IMAP / POP3 command parsers
├── ngx_mail_auth_http_module.c     # external HTTP auth upstream
├── ngx_mail_proxy_module.c         # the actual backend proxy
├── ngx_mail_realip_module.c        # PROXY-protocol-derived client IP
├── ngx_mail_ssl_module.{c,h}       # TLS / STARTTLS for mail
├── ngx_mail_pop3_handler.c         # POP3 protocol handler
├── ngx_mail_pop3_module.{c,h}      # POP3 module
├── ngx_mail_imap_handler.c         # IMAP protocol handler
├── ngx_mail_imap_module.{c,h}      # IMAP module
├── ngx_mail_smtp_handler.c         # SMTP protocol handler
└── ngx_mail_smtp_module.{c,h}      # SMTP module

Key abstractions

Type / function	Role
ngx_mail_session_t	Per-connection state (protocol, auth state, login, password)
ngx_mail_protocol_t	Vtable: init_session, init_protocol, parse_command, auth_state
ngx_mail_init_connection()	Listener handler — accepts and dispatches by configured protocol
ngx_mail_auth_http_init()	Issue the HTTP request to the auth endpoint
ngx_mail_proxy_init()	Open a TCP connection to the chosen backend and start splicing

How it works

sequenceDiagram
    participant Client
    participant Listener
    participant Proto as Protocol handler<br/>(POP3/IMAP/SMTP)
    participant Auth as HTTP auth endpoint
    participant Backend as Mail backend

    Client->>Listener: TCP connect
    Listener->>Proto: dispatch by listener protocol
    Proto->>Client: greeting (e.g., "+OK POP3 ready")
    Client->>Proto: USER alice
    Client->>Proto: PASS s3cret
    Proto->>Auth: HTTP GET / with Auth-User, Auth-Pass headers
    Auth-->>Proto: Auth-Server / Auth-Port + Auth-Status
    Proto->>Backend: connect(Auth-Server:Auth-Port)
    Backend-->>Proto: greeting
    Proto->>Proto: replay USER + PASS to backend
    Proto->>Client: success
    loop traffic
        Client-->>Backend: bytes proxied transparently
        Backend-->>Client: bytes proxied transparently
    end

The flow:

1. Listen — ngx_mail_init_connection (src/mail/ngx_mail_handler.c) is the per-connection init for any mail listener.
2. Greeting — the protocol-specific init_session callback sends the greeting line.
3. Authentication — for POP3 the client sends USER + PASS; for IMAP it sends a tagged LOGIN (or AUTHENTICATE for SASL); for SMTP it goes through EHLO + AUTH. Each protocol's parser populates s->login, s->passwd.
4. HTTP auth lookup — ngx_mail_auth_http_module builds an HTTP/1.0 request to auth_http <url> carrying the client's IP, login, password, and the protocol. The auth endpoint returns headers Auth-Status: OK, Auth-Server: <ip>, Auth-Port: <port> (and optionally a rewritten Auth-User/Auth-Pass).
5. Backend connect — ngx_mail_proxy_module opens a TCP connection to Auth-Server:Auth-Port. If xclient is configured for SMTP, it sends an XCLIENT so the backend sees the original client IP.
6. Splice — the module copies bytes between client and backend until one side closes. There's no mid-stream parsing; once authenticated, it's a raw byte pipe.

Per-protocol details

POP3 (ngx_mail_pop3_*)

• States: START, USER, PASS, AUTH, STAT, CMD, END.
• Supports plain user/pass, APOP (with shared secret), and SASL (AUTH PLAIN, AUTH LOGIN, AUTH CRAM-MD5).
• pop3_capabilities directive controls advertised capabilities.

IMAP (ngx_mail_imap_*)

• Tagged commands; the parser tracks the current tag and produces tagged responses.
• Supports plain LOGIN, AUTHENTICATE PLAIN, AUTHENTICATE LOGIN, AUTHENTICATE CRAM-MD5.
• After login, switches to byte-pipe mode for the rest of the session.

SMTP (ngx_mail_smtp_*)

• Speaks HELO/EHLO, AUTH, MAIL FROM, RCPT TO, DATA.
• Optional XCLIENT injection on backend connect so the backend can see the real client.
• smtp_client_buffer, smtp_greeting_delay tunables for spam-mitigation behaviors.

TLS / STARTTLS

ngx_mail_ssl_module provides:

• listen ... ssl; — direct TLS (POP3S/IMAPS port 995/993, SMTPS 465).
• starttls on; — protocol-level STARTTLS (POP3 STLS, IMAP STARTTLS, SMTP STARTTLS).

Implementation is the standard OpenSSL handshake (see openssl) layered on the connection before the protocol parser starts.

Configuration shape

mail {
    server_name mail.example.com;
    auth_http   http://auth.example.com/cgi-bin/auth;

    pop3_auth   plain apop cram-md5;
    pop3_capabilities "TOP" "USER" "UIDL";

    imap_auth   plain login cram-md5;
    imap_capabilities "IMAP4rev1" "UIDPLUS";

    smtp_auth   plain login cram-md5;
    smtp_capabilities "SIZE 10485760" "STARTTLS";

    server {
        listen     143;
        protocol   imap;
        starttls   on;
        ssl_certificate     /etc/nginx/imap.crt;
        ssl_certificate_key /etc/nginx/imap.key;
    }
}

Integration points

• Event loop — same as HTTP. Each connection has its own ngx_connection_t.
• Configuration system — mail { } is its own block; ngx_mail_module is NGX_CORE_MODULE. Inside, modules of type NGX_MAIL_MODULE register directives.
• OpenSSL — ngx_mail_ssl_module mirrors ngx_http_ssl_module.
• Resolver — used to resolve the auth_http_server's hostname if it's not an IP.

Entry points for modification

The protocol parsers in ngx_mail_parse.c are state-machine hand-written code. SASL mechanism additions go through the per-protocol handler files (ngx_mail_*_handler.c). Auth backend changes go through ngx_mail_auth_http_module.c.

This subsystem sees relatively little churn — the protocols themselves are stable, and most deployments either use this exact feature set or don't use mail proxying at all.