Debugging

Vault is a long-running, concurrent server with a strict crypto-bootstrapping sequence. Debugging it is mostly about getting the logs right and knowing where the locks are.

Server logs

Defaults are usefully verbose. Useful flags:

• -log-level=trace|debug|info|warn|error — global level.
• -log-format=standard|json — JSON for ingestion, standard for humans.
• -log-file plus -log-rotate-bytes, -log-rotate-duration, -log-rotate-max-files — local rotation.
• -combine-logs — write everything to stdout instead of separating server and audit.

Internally, command/log_flags.go translates these flags into a hashicorp/go-hclog logger that flows everywhere through context.

Audit log

Audit devices are also a debugging tool — they record every request and response, with sensitive fields HMAC'd. To pop one up locally:

bin/vault audit enable file file_path=/tmp/vault-audit.log

audit/entry_formatter.go is the template. Vault refuses to start if at least one audit device is configured but unreachable; this is intentional (a silent audit gap is treated as a security incident).

vault operator diagnose

bin/vault operator diagnose runs a battery of checks against a config file and reports which fail (command/operator_diagnose.go, 27k lines). It validates listener TLS, storage connectivity, seal config, service registration, and a number of common misconfigurations.

vault debug

bin/vault debug -duration=2m -interval=10s -output=/tmp/debug.tar.gz snapshots the server's metrics, pprof heap/goroutine/CPU profiles, host info, replication status, and config for a given duration. It's the first thing to attach to a support ticket. Implementation in command/debug.go (34k lines).

Profiling

The HTTP handler exposes pprof under sys/pprof/* (gated behind enable_debug_endpoints). Once enabled:

go tool pprof http://127.0.0.1:8200/v1/sys/pprof/heap
go tool pprof http://127.0.0.1:8200/v1/sys/pprof/profile?seconds=30

vault/logical_system_pprof.go defines the endpoints; access requires sudo capability on the path.

Race conditions and deadlocks

• Run make test (always race-enabled) before assuming code is safe.
• For lock contention, build with the deadlock tag — see vault/test_cluster_detect_deadlock.go. The sasha-s/go-deadlock library will panic with both stacks if a lock is held past the configured threshold.

Common errors and what they mean

Error	Likely cause
Vault is sealed	Pre-unseal request; need vault operator unseal or auto-unseal
permission denied from core.checkToken	Token's policies don't grant the requested capability
local node not active but active cluster node not found	Standby and active disagree about leadership; check storage health
failed to acquire lock	HA storage backend can't grant the leader lock
context deadline exceeded from a backend	A plugin is stuck in a network call; check its logs
mismatched ciphertext in barrier	Storage corruption or wrong key used; usually fatal

Replication / HA debugging

bin/vault status -format=json shows seal status, HA state, and active node addresses. For HA forwarding traces, raise log level on vault.request_forwarding. Replication tools live under vault/replication/cluster.go (CE has stubs only).

UI-side errors

Open the browser devtools and look at the network tab — every failed request returns a JSON body with errors. Mirage (ui/mirage/) lets you reproduce a server response locally without a backend.

When things really go wrong

Vault writes a panic stack to stderr and to the audit log if at all possible. Capture the entire crash log (gist it, don't paste partial fragments) and file an issue with steps to reproduce. The core_metrics.go heartbeat helps establish whether a hang is in storage or the request handler.