Debugging

The Datadog Agent exposes a rich set of introspection surfaces that make debugging in-place practical. Almost every long-running subsystem has a status provider, an expvar registration, and a flare contributor.

Flares

A flare is the canonical way to capture the Agent's state for support escalation or local diagnostics.

./bin/agent/agent flare <case-id>

The flare component (comp/core/flare/) runs every registered flare provider, gathers its output into an in-memory buffer, scrubs known-secret patterns via pkg/redact/, and writes a zip archive to /var/log/datadog (or the equivalent on your platform).

A flare typically contains:

• Configuration (with secrets redacted).
• Recent logs.
• A status snapshot.
• The expvar JSON dump.
• Goroutine and heap profiles.
• Per-component diagnostic output (DogStatsD stats, autodiscovery state, tagger state, network probe state, etc.).

Producing a flare locally is also the fastest way to inspect the running Agent. Internally, the flare command calls into the Agent's IPC API.

status

./bin/agent/agent status

Renders a human-readable summary of the running Agent: aggregator throughput, check states, autodiscovery results, log line counts, etc.

The same data is available as JSON:

./bin/agent/agent status -j

Each component contributes via the status providers registered with comp/core/status/. To add a section, register a Provider returning your component's status.

expvar

The Agent exposes Go's standard expvar package on its API port (default 5002 for the core Agent). With the agent running:

curl -s http://localhost:5002/debug/vars | jq .

This exposes counters, gauges, and other state from across the process. Many internal systems publish via expvar; the comp/core/agenttelemetry/ component re-reads the same data and forwards it to Datadog as agent telemetry.

pprof

The Agent's HTTP server registers net/http/pprof (see imports in cmd/agent/subcommands/run/command.go). On a running agent:

go tool pprof http://localhost:5002/debug/pprof/profile?seconds=30
go tool pprof http://localhost:5002/debug/pprof/heap
go tool pprof http://localhost:5002/debug/pprof/goroutine

Memory-leak debugging guidance lives in docs/public/how-to/memory-profiling/overview.md.

Logs

Agent logs go to stdout/stderr by default and to /var/log/datadog/<binary>.log when running as a service. Log levels are controlled by log_level in datadog.yaml (or DD_LOG_LEVEL).

Log conventions:

• The logging interface is pkg/util/log/ (still being migrated to the comp/core/log/ component).
• Best practices live in docs/public/guidelines/conventions/logging.md. The short version: structured fields where practical, do not log secrets, prefer Errorf-style errors with context.
• Some subsystems have their own loggers (pkg/security/seclog/, pkg/trace/log/) for backwards compatibility.

Health probe

The health probe component (comp/core/healthprobe/) exposes a /live and /ready endpoint. Each subsystem can register itself as healthy/unhealthy. Useful for Kubernetes liveness/readiness probes — and for diagnosing which component is dragging the Agent into an unhealthy state.

curl http://localhost:5555/live
curl http://localhost:5555/ready

(Port comes from health_port in the config.)

diagnose

./bin/agent/agent diagnose

Runs a battery of self-checks (comp/core/diagnose/): Datadog connectivity, port collisions, port firewalls, autodiscovery sanity, etc. Each diagnose suite lives in pkg/diagnose/<area>/.

This is often the first command to run when a customer reports "the agent isn't sending metrics."

connectivity

./bin/agent/agent diagnose connectivity-datadog

Tests reachability of the Datadog backend from the host: TLS handshake, HTTP, latency. Useful for diagnosing firewall and proxy issues.

gohai

./bin/agent/agent gohai

Produces a structured snapshot of the host (OS, kernel, CPU, memory, network interfaces, …). Driven by pkg/gohai/. Datadog also ships this as host metadata.

Common debugging recipes

Symptom	Where to start
"No metrics in Datadog"	agent diagnose connectivity-datadog, agent status (forwarder section), expvar forwarder.* counters
"DogStatsD packets dropped"	agent dogstatsd-stats, expvar dogstatsd.*, raise dogstatsd_buffer_size
"Check is not running"	agent configcheck, agent check <name>, autodiscovery section in agent status
"Trace Agent OOM"	pprof heap, watchdog logs, apm_config.trace_buffer.size
"System Probe not connecting"	system-probe socket path, expvar at system-probe diagnose, kernel feature checks
"CWS rule not firing"	security-agent runtime policy check, SECL evaluator logs

Detailed troubleshooting docs

docs/public/how-to/debug-agents/ contains long-form troubleshooting guides per subsystem. Worth a read once you know which subsystem is misbehaving.

Memory leak hunting

docs/public/how-to/memory-profiling/overview.md walks through the canonical heap-profile workflow. The Agent ships with continuous profiling support (it imports gopkg.in/DataDog/dd-trace-go.v2/profiler in the run command), which can be enabled in datadog.yaml to send profiles to Datadog while the Agent is in the field.