Trace Agent

Active contributors: Gabriel Aszalos, Andrew Glaude, Yang Song, Raphael Gavache, Pierre Gimalac

Purpose

The Trace Agent is Datadog's APM data plane. It receives trace payloads from instrumented applications, normalizes and obfuscates them, decides which traces to keep through several independent samplers, computes RED stats over the full population (independent of sampling), and forwards the result to the Datadog backend.

The bulk of the logic lives in pkg/trace/. The binary entrypoint is cmd/trace-agent/. The Trace Agent can run as a standalone process (the default in containerized deployments) or be embedded inside the core Agent.

A canonical introduction to the trace agent's architecture is the README at pkg/trace/README.md and the diagram in pkg/trace/architecture.png / architecture.puml.

Directory layout

cmd/trace-agent/
├── main.go
├── command/, subcommands/
├── config/              # Trace-agent-specific config
└── internal/

pkg/trace/
├── agent/               # Central orchestrator, processing pipeline
├── api/                 # HTTP/gRPC receiver
├── config/              # Configuration types
├── containertags/       # Async container-tag enrichment
├── event/               # APM event extractor (deprecated)
├── filters/             # Blocklist filters and tag replacement
├── info/                # /info endpoint
├── log/                 # Logger adapter
├── obfuscate/           # SQL/JSON/URL/etc. obfuscators (in pkg/obfuscate, used here)
├── otel/                # gRPC OTLP receiver
├── payload/             # TracerPayloadModifier hook interface
├── pb/                  # Protobuf-generated types
├── remoteconfighandler/ # Apply RC updates to the agent
├── sampler/             # All the samplers
├── semantics/           # Semantic conventions
├── stats/               # Concentrator and client-stats aggregator
├── telemetry/           # Self-telemetry
├── timing/              # Latency timing helper
├── traceutil/           # Span / trace utility helpers
├── transform/           # Payload transformation utilities
├── version/             # Version metadata
├── watchdog/            # CPU/memory watchdog
└── writer/              # Trace writer + Stats writer + shared sender

Pipeline

graph LR
    SDK[Tracer SDK] -->|HTTP v0.3/v0.4/v0.5<br/>UDS or TCP| RECV[Receiver<br/>pkg/trace/api]
    OTEL[OTel SDK] -->|OTLP gRPC| OTLP[OTel Receiver<br/>pkg/trace/otel]
    RECV -->|TracerPayload chan| AGENT[Agent<br/>pkg/trace/agent]
    OTLP --> AGENT
    AGENT --> NORM[Normalize<br/>filter<br/>obfuscate<br/>truncate]
    NORM --> SAMPLERS[Samplers<br/>priority + errors + rare + probabilistic]
    NORM --> CONC[Concentrator<br/>pkg/trace/stats]
    SAMPLERS --> WTRACE[Trace Writer<br/>pkg/trace/writer]
    CONC --> WSTATS[Stats Writer]
    SDK -. precomputed stats .-> CSAGG[Client Stats Aggregator]
    CSAGG --> WSTATS
    WTRACE --> SENDER[Sender]
    WSTATS --> SENDER
    SENDER --> INTAKE[Datadog intake]

The agent fans every trace out to the samplers and the concentrator in parallel. A trace is forwarded if any sampler decides to keep it; the stats system always sees every span regardless of sampling decisions. This is what allows Datadog APM to show accurate request rates and error rates even with heavy sampling.

Samplers

Each sampler is independent. From pkg/trace/sampler/:

Sampler	What it keeps	Notes
Priority sampler	The bulk of traffic, scaled per-service to a target throughput. Respects user-set priorities.	The mechanism by which the agent feeds rates back to clients.
Errors sampler	Traces with error spans (5xx, etc.) regardless of priority.	Rate-limited.
Rare sampler	Newly seen (env, service, op, resource, status, error type) combinations.	Tags spans with _dd.rare.
Probabilistic sampler	Simple rate-based decision, independent.	Off by default.
APM event extractor	Individual spans flagged for Trace Search. Deprecated.	Lives in pkg/trace/event/.

The aggregate keep decision is "any sampler keeps the trace." Sampling rates are returned to clients in the receiver's HTTP response so SDKs can pre-sample at the source.

Stats

Two parallel paths:

• Concentrator (pkg/trace/stats/concentrator.go) — computes RED stats server-side over all received spans. Bucket size defaults to 10 seconds. Aggregation dimensions: service, resource, span name, span type, HTTP status, synthetics flag, peer tags.
• Client stats aggregator (pkg/trace/stats/client_stats_aggregator.go) — accepts pre-computed ClientStatsPayload from newer SDKs, re-aligns timestamps to bucket boundaries, and forwards.

Both feed the Stats Writer which forwards to the backend.

Receiver

pkg/trace/api/ accepts trace payloads in three primary forms:

• MessagePack v0.3, v0.4, v0.5 over HTTP (TCP, UDS, or Windows named pipe).
• gRPC OTLP through pkg/trace/otel/.
• gRPC client stats for SDK-computed stats.

It is rate-limited at the listener and tracks per-language and per-service stats published via /info. After decoding, payloads are pushed onto a Go channel consumed by the Agent.

Obfuscation

Span metadata is obfuscated before forwarding. The obfuscation library lives in pkg/obfuscate/ (a shared package), used here to redact:

• SQL queries (with database-specific dialect support).
• JSON payloads (selective key redaction).
• URLs (query string scrubbing).
• Memcached and Redis commands.

The decision is span-type-driven and configurable from apm_config.obfuscation.

Subcommands

Subcommand	Purpose
trace-agent run	The long-running daemon (default)
trace-agent version	Print version
trace-agent start	Deprecated alias for run
trace-agent test	Helpers under cmd/trace-agent/test/

Configuration

Trace Agent configuration lives under apm_config: in datadog.yaml. Highlights:

• apm_config.enabled — master switch.
• apm_config.receiver_port, receiver_socket — listening endpoints.
• apm_config.max_traces_per_second — priority sampler target.
• apm_config.error_traces_per_second — error sampler limit.
• apm_config.obfuscation.* — obfuscator behavior.
• apm_config.peer_service_aggregation — concentrator dimension toggles.
• apm_config.replace_tags — tag-rewriting rules.

The full schema lives in pkg/trace/config/.

Key abstractions

Type	File	Purpose
Agent	pkg/trace/agent/agent.go	Central orchestrator
Receiver	pkg/trace/api/api.go	HTTP receiver
Concentrator	pkg/trace/stats/concentrator.go	Server-side stats computation
ClientStatsAggregator	pkg/trace/stats/client_stats_aggregator.go	SDK-stats reception
Sampler (interface)	pkg/trace/sampler/sampler.go	Sampler abstraction
PrioritySampler	pkg/trace/sampler/prioritysampler.go	Primary sampler
TraceWriter	pkg/trace/writer/trace.go	Buffered batch sender
StatsWriter	pkg/trace/writer/stats.go	Stats batcher
Sender	pkg/trace/writer/sender.go	HTTP sender with retry/backoff

Entry points for modification

• Adding a new sampler: implement the Sampler interface and register it in the agent's pipeline.
• Adding a new dimension to the concentrator: extend the bucket key in stats/aggregation.go and update payload protos.
• Adding a new obfuscator: extend pkg/obfuscate/ and wire its trigger into the obfuscation pass in agent/obfuscate.go.
• Adding a new receiver protocol version: extend pkg/trace/api/.

Related pages

• Features: APM tracing — end-to-end tracing story.
• Apps: Agent — Trace Agent is launched as a child of the core Agent in many deployments.
• Systems: Forwarder and serializer — how payloads leave the host.