envoyproxy/envoy

Hot restart

Hot restart is Envoy's mechanism for upgrading a running binary without dropping in-flight connections. A new Envoy process starts alongside the old one, takes over the listening sockets, and the old process drains and exits. The protocol and shared-memory layout were described in Matt Klein's hot restart blog post; the implementation lives in source/server/ under the hot_restart_* files.

Purpose

Replace the running binary with a new build with zero downtime.
Migrate listening sockets so the new process accepts connections immediately.
Share key state across the boundary: stat counters (so observability is continuous), drain timing, parent-process metadata.
Preserve in-flight connections on the old process until they close naturally.

Layout

source/server/
├── hot_restart.proto             # The on-the-wire RPC schema
├── hot_restart_impl.{h,cc}       # Production implementation
├── hot_restart_nop_impl.h        # No-op for builds without hot restart
├── hot_restarting_base.{h,cc}    # Shared parent/child plumbing
├── hot_restarting_parent.{h,cc}  # Parent (old) side
├── hot_restarting_child.{h,cc}   # Child (new) side

Key abstractions

Type	File	Role
`Server::HotRestart`	`envoy/server/hot_restart.h`	The interface — used by `InstanceImpl` to negotiate handoff.
`Server::HotRestartImpl`	`hot_restart_impl.h`	Owns the shared-memory region, the Unix domain socket, and the RPC dispatch.
`HotRestartingBase`, `HotRestartingParent`, `HotRestartingChild`	`hot_restarting_*.{h,cc}`	Two ends of the RPC; parent side stays around to serve sockets, child side requests them.

Two-process handoff

sequenceDiagram
    participant Parent as Parent Envoy<br/>(--restart-epoch 0)
    participant Child as Child Envoy<br/>(--restart-epoch 1)

    Parent->>Parent: bind sockets, run
    Note over Parent,Child: Operator launches child

    Child->>Parent: Connect via Unix domain socket
    Child->>Parent: HotRestartVersion RPC
    Parent-->>Child: Compatible
    Child->>Parent: GetParentStats RPC
    Parent-->>Child: Counter values
    Child->>Parent: PassListenSocket(name, addr) per listener
    Parent-->>Child: socket fd via SCM_RIGHTS
    Child->>Child: Bind workers to inherited fds, start accepting
    Child->>Parent: DrainListeners RPC
    Parent->>Parent: Begin drain (drainListeners)
    Note over Parent: drain_time_s elapses
    Child->>Parent: Terminate RPC
    Parent->>Parent: Graceful shutdown

The handoff runs over a Unix domain socket. The address is derived from the --base-id argument plus the --restart-epoch; both processes need the same base ID and consecutive epoch numbers.

Shared memory

The parent process exposes its stat counters in a shared-memory region so the child can pick up where it left off. The layout is versioned so child and parent must speak the same ABI; the version number is exposed via GET /hot_restart_version (admin) and the HotRestartVersion RPC (during handoff). When you change the layout you must bump the ABI; otherwise child startup fails.

HotRestartImpl mmaps a file into the parent's address space; the child mmaps the same file. Counters added by extensions are dynamically mapped through the same machinery.

Socket passing

The PassListenSocket RPC is a SCM_RIGHTS ancillary-message exchange: the parent dups its listening file descriptor and sends it across the Unix domain socket. The child binds it into a Network::ListenSocket and asks workers to accept on it. This is the same mechanism used for in-place listener updates — see listener manager.

For UDP listeners the protocol is the same but the kernel-level socket inheritance handles datagrams differently; see source/common/listener_manager/active_raw_udp_listener_config.cc.

Stages

The lifecycle has three stages, each driven by RPCs from child to parent:

Initialise. Child reads ABI, parent passes per-listener sockets, parent passes accumulated stats.
Drain. Child issues DrainListeners. Parent stops accepting, calls Server::InstanceImpl::drainListeners. The configured drain manager applies its drain time.
Terminate. Child issues Terminate after the drain timeout. Parent shuts down, releases the shared-memory region, exits.

The parent enters drain mode in stage 2: existing connections continue, but new connections go to the child. HTTP/2 GOAWAY and Connection: close headers are added per the drain manager.

Restart epochs

The --restart-epoch flag identifies which generation each process is. The first process is epoch 0, the next 1, the next 2, etc. The Unix domain socket address embeds the epoch so child can target its parent precisely. The Bootstrap.hot_restart_version field can pin to a specific ABI version to prevent accidental cross-version handoffs.

Restarter wrapper

restarter/ at the repo root contains a Python wrapper that operators can use to drive restart automation. It launches a new envoy with the right --base-id / --restart-epoch / --parent-shutdown-time-s and signals the parent.

Failure modes

The protocol is designed to fail closed: if any RPC fails, the child aborts and the parent keeps running. Common failures:

ABI mismatch. The child's compiled-in version doesn't match the parent's shared memory layout. Fix: rebuild parent first, or roll forward both binaries together.
Socket conflict. The child's bootstrap declares listeners the parent doesn't have (or vice versa). The child cannot inherit and binds a new socket; the parent's matching listener still accepts on the old socket until drained.
Drain timeout exceeded. Connections that haven't closed by parent_shutdown_time are forcibly dropped.

Build-time toggles

Hot restart can be disabled at compile time (--define hot_restart=disabled for embedded builds). The no-op implementation in hot_restart_nop_impl.h provides the same interface but does nothing.

Integration points

Server lifecycle. InstanceImpl calls HotRestart::initialize on startup.
Listener manager. Provides the listener fd for PassListenSocket.
Stats. Counter pages live in shared memory.
Drain manager. Drives the parent's drain when DrainListeners is received.
Admin. POST /drain_listeners issues the same drain locally; GET /hot_restart_version reports the ABI.

Entry points for modification

Modifying the RPC protocol: edit hot_restart.proto and bump the version.
Modifying the shared-memory layout: same — bump version.
Modifying drain semantics: source/server/drain_manager_impl.cc.