Debugging

Practical notes on triaging the most common kinds of failures.

Where logs go

All Rust services use tracing with a JSON formatter from libs/tracing-utils/. By default each binary writes to stderr. In a cargo neon workflow logs are captured per-service inside .neon/:

• .neon/pageserver.log
• .neon/safekeepers/sk1/safekeeper.log
• .neon/storage_broker.log
• .neon/endpoints/<name>/compute_ctl-???.log
• .neon/endpoints/<name>/postgres.log

The RUST_LOG env var is honored. Common settings:

RUST_LOG=info,pageserver=debug cargo neon start
RUST_LOG=trace cargo neon endpoint start main    # very chatty

LOGFMT=text switches the JSON formatter to a human-readable line format and is useful when running a single binary by hand.

Reading a tracing span tree

Each request gets a span chain like tenant{id=…} timeline{id=…} get_page_at_lsn{key=… lsn=…}. When debugging a failed GetPage@LSN, search the pageserver log for the request_id from the compute side and follow the spans down to the layer-map lookup and reconstruct call.

The debug_endpoints (pageserver) HTTP API also exposes per-tenant per-timeline runtime state — see pageserver/src/http/routes.rs and pageserver/src/tenant/debug.rs.

Common failure modes

The compute won't start

The compute_ctl log (.neon/endpoints/<name>/compute_ctl-*.log) is the first place to look. It contains:

• The spec it received (URLs of pageservers, safekeepers, list of extensions).
• The basebackup download progress.
• The postgres start command.
• Any errors from running pre-start SQL.

If the compute starts but immediately exits, look at postgres.log. A common cause is a corrupted basebackup, which usually points back to an issue in the pageserver.

GetPage@LSN is slow or stuck

Three places to look:

1. Pageserver layer map. A timeline with too many L0 layers piles up the redo chain. The pageserver_layer_count metric is a good first signal. Compaction (see pageserver/src/tenant/timeline/compaction.rs) is what reduces this.
2. WAL-redo process health. Each tenant has a long-lived postgres --wal-redo subprocess. If it's stuck, all GetPage@LSN for that tenant stall. The pageserver_walredo_* metrics surface this.
3. Remote storage download stalls. Layer files are downloaded on demand. Slow remote storage (S3, GCS, Azure) shows up as long-held read latencies. The remote_timeline_client metrics in pageserver/src/tenant/remote_timeline_client.rs cover this.

A safekeeper went read-only

Safekeepers refuse new writes when they fall out of quorum. Look in the safekeeper log for flush_lsn falling behind, and check the safekeeper_election_* metrics. The safekeeper/spec/ TLA+ spec is the ground truth for what "should" happen.

A test fails, what now?

Rerun with --basetemp=/tmp/pytest --preserve-database-files, then attach to the leftover .neon working directory:

cargo neon stop --working-dir /tmp/pytest/.neon
# ...edit logs, rerun bits manually, attach gdb to a stuck postgres compute, etc.

Most fixtures honor NEON_BIN / POSTGRES_DISTRIB_DIR for swapping in custom builds.

Attaching gdb / lldb

The compute is a normal Postgres process. Find its PID from cargo neon endpoint list or ps, then:

sudo gdb -p <pid>
(gdb) bt

The pageserver and safekeeper are Rust binaries with full debug info in debug builds and line-only info in release-line-debug profile (Cargo.toml defines several release-line-debug-* profiles for production).

Flamegraphs

README.md recommends flamegraph-rs. One subtle requirement:

If you're using lld or mold, you need the --no-rosegment linker argument.

This is a general Rust + lld/mold issue, not Neon-specific. See PR #6764 for details.

cargo flamegraph -p pageserver --bin pageserver -- run --config-file ...

The pageserver also has built-in pprof support via jemalloc_pprof (see proxy/src/jemalloc.rs and the analogous code in pageserver). Hitting /debug/pprof/heap returns a heap profile.

Failpoints

Tests can inject failures using the fail crate. The pageserver/tests/ and test_runner/regress/ directories have many examples. To exercise a failpoint manually:

curl -X PUT 'http://localhost:9898/v1/failpoints' \
  -H 'Content-Type: application/json' \
  -d '[{"name": "after-timeline-load", "actions": "return"}]'

The full list of named failpoints is found via git grep 'fail_point!' in the codebase.

Storage-side debug tools

• pagectl (pageserver/ctl/) — offline tool for inspecting layer files: pagectl layer list, pagectl layer dump, etc.
• storcon-cli (control_plane/storcon_cli/) — talks to the storage controller's API to list tenant placement, request migrations, etc.
• safekeeper/client/ and pageserver/client/ — Rust clients you can pull into ad-hoc binaries when scripting against a running stack.

Symbolicating panics

Panics include backtraces by default (anyhow is the workspace error type and pulls backtrace). For minidumps from production, the Sentry crate (sentry = … in Cargo.toml) can be configured via env var:

SENTRY_DSN=... cargo run -p pageserver

In release builds debug info is on (debug = true in [profile.release] in Cargo.toml) so backtraces resolve symbols cleanly.

See also

• How to monitor / Logging — log conventions in production.
• Patterns and conventions — error-handling style (anyhow, thiserror).
• Pageserver — internal model for layer files and timelines.