envoyproxy/envoy

Debugging

A grab bag of techniques for working out what a running Envoy is doing — when reading code, when running tests, and when looking at a production process.

Logs

Envoy uses spdlog under the hood. Components are registered in source/common/common/logger.h; a few of the busiest:

connection, client, pool
http, http2, quic
router, filter, ext_authz
upstream, cluster, health_checker
config, grpc, runtime, secret
admin, main, server

Set log levels at startup or via the admin interface:

# At startup
envoy -c bootstrap.yaml --log-level debug
envoy -c bootstrap.yaml --component-log-level upstream:trace,http:debug

# At runtime via admin
curl -X POST 'localhost:9901/logging?level=debug'
curl -X POST 'localhost:9901/logging?paths=upstream:trace,http:debug'

Trace-level logging in the data plane is voluminous and slow; turn it on for a single component when debugging a specific path.

ENVOY_LOG_* macros wrap spdlog and add the component name; new code should always use them. See source/common/common/logger.h.

The admin interface

The admin server (port 9901 by default, configurable via Bootstrap.admin) is the operator's debug console. The most useful endpoints when debugging code:

Endpoint	What it shows
`GET /server_info`	Version, build flags, uptime, state
`GET /stats`	Every counter, gauge, histogram
`GET /stats?format=prometheus`	Same in Prometheus format
`GET /stats?usedonly`	Stats with non-zero values
`GET /listeners`	Configured listeners and bound addresses
`GET /clusters`	Clusters, hosts, health, traffic, conn pools
`GET /config_dump`	Live `Bootstrap`, `LDS`, `CDS`, `RDS`, `SDS`, `RTDS` view
`GET /config_dump?include_eds`	Include EDS endpoints (large)
`GET /runtime`	Runtime keys/values
`POST /runtime_modify?key=value`	Override a runtime key
`GET /memory`	tcmalloc / mimalloc heap stats
`GET /heapprofiler`	Start/stop tcmalloc heap profiling (build-time gated)
`GET /contention`	Mutex contention (build-time gated)
`GET /hot_restart_version`	Hot-restart RPC ABI
`POST /healthcheck/fail` / `/healthcheck/ok`	Toggle health check failure (when enabled)
`POST /reset_counters`	Zero out counters
`POST /quitquitquit`	Graceful shutdown

The implementation lives in source/server/admin/. Each handler is a *_handler.cc/.h pair.

If Envoy crashes (or you SIGABRT it) the binary prints a backtrace via source/common/signal/signal_action.cc and source/server/backtrace.h. The frames are addresses; resolve them to file/line with the symbol decoder script:

tools/stack_decode.py /path/to/envoy-static < crash.log

tools/stack_decode.py reads the binary's symbols (with addr2line or llvm-symbolizer) and rewrites the addresses in place.

gdb, lldb, rr

Build with -c dbg for full debug info. The binary is statically linked, so you can gdb ./bazel-bin/source/exe/envoy-static directly.

Useful breakpoints when working on the data plane:

Envoy::Http::ConnectionManagerImpl::onData — every byte that hits HTTP from a downstream connection.
Envoy::Router::Filter::decodeHeaders — every routed request.
Envoy::Network::ConnectionImpl::onReadReady — every readable event on a TCP connection.
Envoy::Server::WorkerImpl::start — worker thread bootstrap.
Envoy::ThreadLocal::SlotImpl::set — main → workers config propagation.

For determinism, rr works well with Envoy in -c dbg builds and is invaluable for races.

ASan / TSan / MSan / UBSan

The fastest way to find a memory or threading bug is to run the relevant test under a sanitizer:

bazel test --config=asan  //test/common/http/...
bazel test --config=tsan  //test/common/upstream/...
bazel test --config=msan  //test/common/network/...
bazel test --config=ubsan //test/common/...

CI runs the whole tree under each sanitizer for every PR; if your local test passes but CI fails on a sanitizer, run the same target locally with --config=asan (or whichever).

Watchdog

The GuardDog thread monitors worker event loop liveness and will kill the process if a loop is stuck longer than miss/megamiss thresholds. When debugging a hang, configure higher thresholds in Bootstrap.watchdogs or attach gdb before the watchdog fires.

The watchdog can be configured to take an extension action (write a diagnostic, dump cores, etc.) via source/extensions/watchdog/profile_action/.

Performance and profiling

CPU profiling. Build with --define tcmalloc=gperftools and use bazel/PPROF.md. The admin endpoint /cpuprofiler toggles profiling on/off.
Heap profiling. Same toolchain via the admin /heapprofiler endpoint.
Counters. The admin /stats?usedonly view is the fastest way to see what's happening at L4/L7. The Envoy stat hierarchy is documented in source/docs/stats.md.
Tracing. Add a tracer to the bootstrap (e.g. OpenTelemetry) and inspect spans in your tracing UI.

Common gotchas

State sharing across workers. If a value is the same on every worker only by accident, you have a thread-local slot bug waiting to happen. Always go through ThreadLocal::SlotImpl for shared state.
Filter state in the wrong direction. Decoder filters see request data; encoder filters see response data. The filter manager (source/common/http/filter_manager.cc) treats them as separate chains; data does not magically appear in the other direction.
Holding raw pointers across event loop iterations. Anything that lives on a connection can be deferred-deleted at the end of the iteration. Hold a std::shared_ptr or weak handle if you need to outlive the call stack.
Forgetting cancel() on async clients. source/common/http/async_client_impl.cc returns request handles you must cancel if your filter is being destroyed before the response.