troubleshoot

The github.com/hashicorp/consul/troubleshoot module contains the libraries that power the consul troubleshoot CLI and a few external helpers shipped with consul-k8s. It is the rare "user-facing diagnostics" library that doesn't depend on the agent — it talks to a running agent over the public APIs.

Sub-packages

Path	Purpose
troubleshoot/proxy/	Inspect Envoy admin API and Consul HTTP to diagnose a sidecar's wiring
troubleshoot/ports/	TCP probes for the well-known Consul ports (8300, 8301, 8302, 8500, 8501, 8502, 8600)
troubleshoot/validate/	Validate Envoy upstream configuration against Consul's expected state

The matching CLI commands live in command/troubleshoot/:

• consul troubleshoot proxy — calls troubleshoot/proxy/
• consul troubleshoot upstreams — also calls troubleshoot/proxy/
• consul troubleshoot ports — calls troubleshoot/ports/

Diagnostic flow

graph LR
    Op[Operator] --> CLI[consul troubleshoot proxy]
    CLI --> Lib[troubleshoot/proxy]
    Lib --> Envoy[Envoy admin :19000]
    Lib --> Consul[Consul HTTP :8500]
    Lib -->|comparison| Report[Findings + suggested actions]
    Report --> Op

The library fetches:

• The list of services the proxy claims (Envoy /clusters, /listeners, /config_dump).
• The set of services Consul thinks should be reachable (/v1/health/connect/<svc>).
• Intentions matching the proxy's identity.

It diffs them and surfaces specific failure modes:

• "Envoy doesn't see service X but it should" → catalog connectivity / xDS issue.
• "Service X is unhealthy in Consul" → health check failure.
• "Intention denies traffic from this proxy to X" → ACL / intention misconfiguration.
• "TLS certificate expired" → CA / leaf rotation problem.

Each finding includes a remediation hint pointing the operator at the right config entry, command, or doc.

Module boundary

troubleshoot is a small public library. It depends on api/ for Consul HTTP calls and on Envoy admin endpoints (which are fetched by hand-rolled HTTP because there is no Envoy admin Go SDK).

Tests

troubleshoot/proxy/*_test.go and friends use fixture JSON dumps from real Envoy admin endpoints. They live next to the test files; check them in alongside any change.

Entry points for modification

• Add a new diagnostic check: write a function in troubleshoot/proxy/ that returns []Finding. Hook it into the dispatcher in troubleshoot/proxy/proxy.go.
• Add a new top-level command: scaffold under command/troubleshoot/<x>/ and call into the library function.
• Add a new port to the port-check list: troubleshoot/ports/.