cloudflare/pingora

Graceful upgrade

Pingora can replace a running binary with a new one and not drop a single connection. This works on Linux only.

The recipe is:

pkill -SIGQUIT load_balancer
RUST_LOG=INFO ./load_balancer -c conf.yaml -d -u

The old process gets SIGQUIT; the new process is started with --upgrade (-u). The new process opens a Unix-domain socket at the path configured by upgrade_sock:, the old process connects to it and sends its listening file descriptors, then the new process starts accepting on those FDs. The old process keeps running until in-flight connections finish.

The transfer protocol

sequenceDiagram
    participant Old as Old process
    participant Sock as upgrade_sock<br/>(Unix domain)
    participant New as New process

    Note over Old: Receives SIGQUIT
    Note over New: Started with -u
    New->>Sock: bind, listen
    Old->>Sock: connect
    Old->>Sock: SCM_RIGHTS msg<br/>with listen FDs
    Sock->>New: deliver FDs
    New->>New: build TransportStacks<br/>around received FDs
    New->>New: start accepting
    Note over Old: Stop accepting<br/>(close listen sockets)
    Note over Old: Wait for in-flight<br/>requests (EXIT_TIMEOUT = 300s)
    Note over Old: Exit

The wire protocol is implemented in pingora-core/src/server/transfer_fd/. It uses Unix-domain SCM_RIGHTS to send file descriptors out-of-band. Each FD is paired with the endpoint string it belongs to (e.g. 0.0.0.0:6188) so the new process can match them up against its configured listeners.

Server-side state machine

The phases (from ExecutionPhase in pingora-core/src/server/mod.rs):

Phase	What happens
`Bootstrap`	New process: try to connect to `upgrade_sock`. If it succeeds, receive FDs from old process. Otherwise just bind fresh.
`BootstrapComplete`	FDs ready (either from upgrade or fresh)
`Running`	Serving traffic
`GracefulUpgradeTransferringFds`	Old process: SIGQUIT received; listening on `upgrade_sock`, sending FDs to new process
`GracefulUpgradeCloseTimeout`	Old process: FDs sent; waiting `CLOSE_TIMEOUT` (5s) for new process to take over
`GracefulTerminate`	Both processes: drain in-flight
`ShutdownStarted` / `ShutdownGracePeriod` / `ShutdownRuntimes`	Old process winding down

The 300-second exit timeout is EXIT_TIMEOUT in pingora-core/src/server/mod.rs. The 5-second close timeout is CLOSE_TIMEOUT. Both are constants — to change them you patch the source.

The signal protocol

Signal	Effect
`SIGTERM`	Graceful shutdown. Stop accepting new connections, drain in-flight, exit after `EXIT_TIMEOUT` or when drained.
`SIGQUIT`	Graceful upgrade. Wait for new process to take FDs, then drain.
`SIGINT`	Quick shutdown — close everything immediately.
`SIGUSR1`	Sent by the new daemon to the old parent to signal "I'm ready". Optional, gated by `daemon_wait_for_ready` config key.

Daemon coordination signals

Added in commit ee387f4 (Mar 2026): a fuller signaling mechanism between old and new processes during graceful upgrade. The old process can wait on a readiness signal (SIGUSR1) before backing out, which prevents systemd from racing the SIGQUIT.

The daemon_wait_for_ready, daemon_ready_timeout_seconds, and daemon_notify_timeout_seconds config keys control this. See Configuration.

Edge cases

Different listen sets: if the new process's configuration listens on different addresses than the old one, only the matching FDs transfer. The new process binds the rest fresh; the old process's other listeners go away when it exits.
Wrong upgrade_sock path: the new process can't find the old; it binds fresh and starts accepting in parallel. Now you have two processes serving the same listen address — the OS demultiplexes by SO_REUSEPORT or you get connection failures. Always use the same upgrade_sock value.
Permission errors after fork: when the daemon drops UID, there's a brief window where the parent can't be signaled by the daemon. daemon_notify_timeout_seconds (default 60) covers this.
Listen FDs not inherited: the bug in b633683 (Mar 2026) was that bootstrap_as_a_service didn't inherit listen FDs through graceful upgrade for service-with-dependents shapes. Fixed.

Source map

Concern	File
Top-level driver (`run_forever`)	`pingora-core/src/server/mod.rs`
FD-passing protocol	`pingora-core/src/server/transfer_fd/`
Daemonization (and SIGUSR1)	`pingora-core/src/server/daemon.rs`
`Opt::upgrade` (`-u` flag)	`pingora-core/src/server/configuration/mod.rs`
Listener-side acceptance of inherited FDs	`pingora-core/src/listeners/mod.rs`