Open-Source Wikis

/

Pingora

/

How to contribute

/

Debugging

cloudflare/pingora

Debugging

Turn on logs

Pingora uses the log crate. Set RUST_LOG:

RUST_LOG=trace cargo run --features "proxy lb openssl"
RUST_LOG=pingora_proxy=trace,pingora_cache=debug cargo run ...

The error log normally goes to STDERR. If error_log: is set in the YAML config, it goes there instead. When daemonized, STDERR is redirected to the configured error log file. See docs/user_guide/error_log.md.

Common errors

502 Bad Gateway on every request

Usually one of:

  • Your upstream_peer() returned a peer the proxy can't connect to.
  • The TLS SNI doesn't match the upstream's certificate. HttpPeer::new(addr, true, sni) — the third argument is the SNI string.
  • A request_filter returned Err, and the default error response is being sent. Check the error log.

fail_to_connect() is being called repeatedly

Pingora retries when e.retry() is true on the returned error. If you want to give up after one failed attempt, set e.set_retry(false) inside fail_to_connect(). See docs/user_guide/failover.md.

connection reset by peer on the upstream side

Often a TLS issue. Confirm the SNI in your HttpPeer. If you're using pingora-rustls, double-check that the feature warning ("highly experimental") is acceptable for your scenario.

Daemon won't restart after upgrade

If daemon_wait_for_ready is true in the YAML config, the parent process waits for SIGUSR1 from the new daemon. Confirm:

  • daemon_ready_timeout_seconds is large enough (default 600).
  • The new process has permission to send SIGUSR1 to the parent. If the daemon drops UID after fork, there's a brief window where the parent's UID and daemon's UID disagree. daemon_notify_timeout_seconds (default 60) controls the retry budget for that window.

Cache is silently disabled

HttpCache::phase becomes Disabled(NoCacheReason). Read the reason. Common ones: NeverEnabled (you didn't call enable_cache_filter), BypassReason::* (your cache_lookup_filter returned bypass), or storage error. Use pingora-cache/src/trace.rs spans (gated by the trace feature) for cache phase tracing.

Sentry

The sentry cargo feature wires pingora-core into sentry-rust. Panics, errors, and a chunk of contextual info get reported to your Sentry DSN. Useful in production. Configure via Server::configuration.sentry (see pingora-core/src/server/configuration/).

If you ran the daemon and Sentry stopped reporting after daemonize, that was a real bug — fixed in 9a4eee3 ("Reinit sentry after daemonize", Mar 2026).

Panics

By default, panics kill the worker thread but the server keeps running. The server reports panics via Sentry (if enabled) and falls through. See docs/user_guide/panic.md for the exact behavior. Pingora deliberately doesn't catch_unwind in the request loop — it relies on tokio's task-level isolation.

Reading the proxy state machine

When something goes wrong inside proxy_h1.rs or proxy_h2.rs, the most useful breakpoints are:

  • HttpProxy::proxy_to_upstream (in proxy_h1.rs / proxy_h2.rs) — main proxy loop
  • HttpProxy::handle_cache_* (in proxy_cache.rs) — cache state machine transitions
  • ProxyHttp::fail_to_proxy — the catch-all error sink

debug! and trace! calls are sprinkled throughout these files. Search for error! and warn! first to find the user-actionable log messages.

Debugging the connection pool

pingora-pool/src/connection.rs keeps an in-memory pool of upstream connections per peer. The unexpected data counter (added in ea9d9ec, Mar 2026) tracks how often a pooled connection had unexpected bytes when checked out — usually a sign of a broken upstream or an HTTP/2 protocol violation. Watch this metric in production.

Debugging health checks

pingora-load-balancing/src/health_check.rs spins a background task per backend. If a backend is unexpectedly marked unhealthy, log the health-check failure reason — TCP-level checks log connection errors, HTTP-level checks log the response status.

Useful environment variables

Var Effect
RUST_LOG Log filtering (e.g. pingora_proxy=trace)
RUST_BACKTRACE=1 Panic backtraces
SSLKEYLOGFILE Wireshark-compatible TLS keylog when using pingora-rustls

Built by Factory AutoWiki from public repository content. It is a generated preview for codebase exploration, not source-maintained documentation.

Debugging – Pingora wiki | Factory