APM tracing

Purpose

Datadog APM lets engineers instrument their applications, view distributed traces, and analyze RED stats per service. The Agent side of APM is the Trace Agent — it accepts trace payloads from instrumented applications, makes sampling decisions, computes RED stats, and forwards both to the Datadog backend.

This page is the end-to-end story. For the Trace Agent binary's internals, see Apps: Trace Agent. For the broader APM context (UI, query language, libraries), refer to docs.datadoghq.com.

Architecture

graph LR
    APP1[App<br/>dd-trace-go / py / java / ...] -->|MessagePack v0.4 / v0.5| TRA[Trace Agent receiver]
    APP2[App<br/>OpenTelemetry SDK] -->|OTLP gRPC| OTLP[OTLP receiver]
    TRA -->|TracerPayload chan| AGENT[Trace Agent processing pipeline]
    OTLP --> AGENT
    AGENT -->|all spans| CONC[Concentrator<br/>RED stats]
    AGENT -->|sampler input| SAMP[Samplers]
    SAMP -->|kept traces| TWRITE[Trace Writer]
    APP1 -.->|client-computed stats| TRA
    TRA -->|client stats| CSAGG[Client Stats Aggregator]
    CSAGG --> SWRITE[Stats Writer]
    CONC --> SWRITE
    TWRITE --> SENDER[Sender]
    SWRITE --> SENDER
    SENDER --> INTAKE[Datadog APM intake]

Note that stats are computed over every span — sampling does not affect them. The backend can therefore show accurate request counts and error rates even when only 1% of traces are kept.

Reception

The trace agent listens on:

• HTTP TCP (default 127.0.0.1:8126) — the canonical path for the Datadog tracing libraries.
• Unix Domain Socket — preferred in containerized deployments to avoid networking namespace concerns.
• Windows named pipe.
• gRPC OTLP via pkg/trace/otel/.

Multiple tracer-protocol versions coexist (v0.3 / v0.4 / v0.5). v0.5 is the most efficient: it uses a string interning table to avoid repeating service names, span names, and tag keys.

Processing pipeline

pkg/trace/agent/ runs every received trace through a sequential pipeline:

1. Normalization — defaults, truncation, validation. Bad traces are rejected.
2. Filtering — apm_config.filter_tags, apm_config.ignore_resources.
3. Tag injection — globally configured tags merged into every span.
4. Obfuscation — SQL queries, Redis/Memcached commands, JSON payloads, URLs. Per-span-type strategies.
5. Truncation — caps oversized fields.
6. Top-level span detection — used by the concentrator and samplers.
7. Tag replacement rules — apm_config.replace_tags regex rewriters.

Then the trace fans out to the samplers and the concentrator in parallel.

Sampling

Multiple independent samplers run; a trace is kept if any of them decides to keep it. From pkg/trace/sampler/:

• Priority sampler — the primary mechanism. Maintains per-(service, env) target throughput; respects user-set priorities (-1 drop, 0 auto-drop, 1 auto-keep, 2 user keep). The computed rate is fed back to the SDK in the receiver's HTTP response so the SDK can pre-sample.
• Errors sampler — keeps traces with error spans up to a rate cap.
• Rare sampler — keeps the first occurrence of new (env, service, op, resource, status, error type) combinations.
• Probabilistic sampler — opt-in flat rate.
• APM event extractor (deprecated).

Configuration lives under apm_config.sampler.*, apm_config.error_sampler.*, apm_config.rare_sampler.*, apm_config.probabilistic_sampler.*.

Stats

Two parallel stats paths:

• Concentrator (pkg/trace/stats/concentrator.go). Computes RED stats server-side. Aggregation buckets (default 10 s) are keyed on (service, resource, span name, span type, HTTP status, synthetics, peer tags). At end of bucket, emits a StatsPayload to the Stats Writer.
• Client stats aggregator (pkg/trace/stats/client_stats_aggregator.go). Newer SDKs compute their own stats and submit them via a separate endpoint. The aggregator re-aligns timestamps and forwards to the Stats Writer.

Stats see every span regardless of sampling. This is why APM dashboards stay accurate at high sampling rates.

Obfuscation

pkg/obfuscate/ (a shared package) is heavy on detail because protecting customer data is critical:

• SQL — dialect-aware redaction of literals while preserving structure.
• JSON — selective key redaction.
• URL — query string scrubbing.
• Redis / Memcached / DDB / SQS — command-specific scrubbers.

Configuration: apm_config.obfuscation.*.

Tracer SDKs

Tracer SDKs live in language-specific repositories (e.g., dd-trace-go, dd-trace-py). They share a common protocol with the Trace Agent.

The Datadog Agent itself is a tracer client too: cmd/agent/subcommands/run/command.go imports gopkg.in/DataDog/dd-trace-go.v2/profiler and submits its own traces and profiles via the local Trace Agent.

Auto-instrumentation

The Cluster Agent's admission controller can mutate pods on creation to inject the right Datadog tracer library and set DD_TRACE_AGENT_URL to point at the Agent's UDS or TCP endpoint. Language detection (pkg/clusteragent/languagedetection/) decides which SDK to inject.

Universal Service Monitoring vs. APM

NPM's USM (pkg/network/usm/) decodes application protocols inside the kernel to produce service-level metrics without instrumentation. APM, by contrast, requires SDK instrumentation but produces full request-level traces. The two systems coexist and complement each other; they do not share a code path.

Internal observability

• agent status includes a Trace Agent section.
• The Trace Agent exposes /info with per-language receiver stats.
• Internal trace-agent metrics flow back into the Datadog account as datadog.trace_agent.*.
• An optional pprof endpoint runs on the Trace Agent.

Configuration knobs

The most-touched ones:

Key	Effect
apm_config.enabled	Master switch
apm_config.receiver_port, receiver_socket	Reception surface
apm_config.max_traces_per_second	Priority sampler target
apm_config.error_traces_per_second	Errors sampler limit
apm_config.obfuscation.*	Obfuscator behaviour
apm_config.peer_service_aggregation	Concentrator dimension toggles
apm_config.replace_tags	Tag rewriting
apm_config.compute_stats_by_span_kind	OTel span-kind-aware stats
apm_config.dd_url	Override APM intake URL

Entry points for modification

• New sampler: implement Sampler and register it in the agent's pipeline.
• New OTLP attribute mapping: extend pkg/trace/otel/.
• New obfuscation strategy: extend pkg/obfuscate/.
• New protocol version: extend pkg/trace/api/.

Related pages

• Apps: Trace Agent — the binary internals.
• Apps: OTel Agent — the OTel-first variant.
• Systems: Forwarder and serializer — what comes after the trace writer.