Metrics pipeline

Purpose

The metrics pipeline is the path a metric takes from its origin (a DogStatsD packet, a Python check, a Go core check) to the Datadog intake. This page traces the path end-to-end and points at the relevant detail pages.

End-to-end view

graph LR
    APP1[App<br/>statsd client] -->|UDP/UDS/pipe| DSD[DogStatsD listener]
    APP2[App<br/>OpenMetrics] -->|scrape| OPENM[OpenMetrics check]
    PCHECK[Python check] -->|Sender API| AGG
    GCHECK[Go core check] -->|Sender API| AGG
    DSD -->|MetricSample| AGG[Aggregator]
    OPENM -->|Sender API| AGG
    AGG -->|Series + Sketches + Events + Service checks| SER[Serializer]
    SER -->|protobuf / JSON| FWD[Forwarder]
    FWD -->|HTTP POST<br/>/api/v2/series<br/>/api/beta/sketches| INTAKE[Datadog intake]

Sources

A metric can enter the pipeline through several front doors:

• DogStatsD — UDP, Unix Datagram Socket, Unix Stream Socket, or Windows named pipe. See Apps: DogStatsD. Origin tags are inferred from the source PID's cgroup, an explicit c:<container_id> tag, or the unified-origin-detection binary header.
• Python checks — the integration ecosystem (datadog-agent integration packages and the bundled checks). They use the AgentCheck Python class, which calls back into the Go aggregator through cgo via rtloader.
• Go core checks — checks compiled into the Agent (CPU, memory, container runtimes, network, eBPF-driven OS metrics, JMX bridge).
• OpenMetrics check — pulls Prometheus-style metrics from arbitrary HTTP endpoints. It is a Python check that delegates to a Go-implemented scraper for performance.
• Cluster Checks Runner — when running in Kubernetes, cluster-level checks are dispatched from the Cluster Agent. See Apps: Cluster Agent.
• Custom metrics from External Metrics Provider — also via the Cluster Agent.
• OTLP metrics — the Agent has an OTLP receiver (comp/otelcol/); OTLP metrics map to MetricSamples through a translation layer.

Aggregator

Once a sample is in the agent process, the path is the same regardless of source:

1. Demultiplexer receives the sample (pkg/aggregator/demultiplexer*.go).
2. Context resolver maps (name, tags, host) to a stable internal context key.
3. TimeSampler (DogStatsD) or CheckSampler (checks) aggregates samples into per-context state.
4. On the flush tick, the aggregator emits Series and SketchSeries lists, plus any pending events and service checks.

Detail: Aggregator pipeline.

Serializer

The aggregator's output is handed to the serializer (pkg/serializer/, comp/serializer/):

• Series are encoded as protobuf (preferred) or JSON.
• Sketches are encoded as protobuf with embedded DDSketch payloads.
• Events and service checks are encoded as JSON.
• All payloads are zstd-compressed if compression is enabled.
• Payloads exceeding the per-request size limit are split into multiple HTTP requests.

Detail: Forwarder and serializer.

Forwarder

The default forwarder (comp/forwarder/defaultforwarder/):

• Authenticates using the configured API key(s).
• Sends to the primary intake (dd_url) and any additional_endpoints.
• Retries on failure with exponential backoff. In-memory queue first; on-disk queue when forwarder_storage_path is configured.
• Supports Multi-Region Failover (MRF).

A separate event-platform forwarder (comp/forwarder/eventplatform/) ships logs, CWS events, dyninst payloads, and a few other non-metric streams.

Intake endpoints

Endpoint	Payload
/api/v2/series	Series
/api/beta/sketches	Sketches
/api/v1/check_run	Service checks
/intake/?api_key=...	Events and host metadata
/api/v2/logs	Logs (via event platform forwarder)

Internal telemetry

The pipeline is heavily self-instrumented. Every stage exposes counters via expvar and feeds them into Agent telemetry. Notable internal metrics:

• dogstatsd.processed, dogstatsd.packets_received, dogstatsd.packet_pool_*
• aggregator.flush.*, aggregator.context_*, aggregator.sketch_*
• serializer.*
• forwarder.transactions.*, forwarder.api_key_status.*

agent dogstatsd-stats and agent status are the operator-facing surfaces.

DogStatsD performance

DogStatsD is one of the more performance-sensitive subsystems. Several optimizations are visible:

• A bounded packet pool to keep allocations low.
• A worker pool for parsing.
• Tag canonicalization done once at receive time, cached by hash.
• A "no-aggregation" mode that streams samples through unchanged for use cases that need every observation.
• Capture/replay (comp/dogstatsd/replay/) for benchmarking and debugging.

OTLP metrics path

The Agent exposes an OTLP receiver via comp/otelcol/. OTLP metric payloads are translated to Datadog-native metrics via pkg/opentelemetry-mapping-go. The translated metrics enter the same aggregator pipeline.

Tags everywhere

Every metric carries tags. The tagger (see Workloadmeta and tagger) is the canonical source. DogStatsD origin detection uses it; checks use it via the Sender API; the OTLP receiver uses it to apply environment tags. This is what makes per-container tagging work consistently across all metric sources without each source re-implementing it.

Configuration knobs

The most-touched ones:

Key	Effect
api_key, dd_url, additional_endpoints, mrf_dd_url	Backend addressing
dogstatsd_*	DogStatsD listeners and pipeline
histogram_aggregates, histogram_percentiles	Histogram aggregation
flush_interval	Sampler flush cadence
aggregator_buffer_size	Internal channel sizing
forwarder_*	Retry and disk queue

Entry points for modification

• New metric source: implement a Sender consumer or an aggregator entry point. Most new sources should go through the existing Sender API.
• New metric type: extend pkg/metrics/, then aggregator handlers, then serializer encoders.
• New endpoint or payload format: extend the serializer and a forwarder variant.

Related pages

• Apps: DogStatsD
• Systems: Aggregator pipeline
• Systems: Forwarder and serializer
• Systems: Check runtime
• Systems: Workloadmeta and tagger