Aggregator pipeline

Active contributors: Olivier G, Vickenty Fesunov, Rémy Mathieu

Purpose

The aggregator is the heart of the Agent's metric pipeline. Every metric that leaves the host — whether it came from DogStatsD, a Python check, or a Go core check — flows through pkg/aggregator/, gets sampled and aggregated according to its type, and emerges as a Series or SketchSeries ready for the serializer.

The package is one of the longest-lived in the repo (its README in pkg/aggregator/README.md predates the move to comp/), and its conceptual shape — DogStatsD vs. checks → TimeSampler / CheckSampler → ContextMetrics → Serializer — has been stable since Agent v6.

Pipeline overview

graph LR
    DSD[DogStatsD] -->|MetricSample| AGG[Aggregator]
    SENDER[Sender<br/>used by checks] -->|MetricSample| AGG
    AGG -->|by source| TS[TimeSampler<br/>DogStatsD] & CS[CheckSampler<br/>per check instance]
    TS --> CTX[ContextMetrics]
    CS --> CTX
    CTX --> SER[Serializer]
    AGG -->|Events / ServiceCheck| ES[Event/ServiceCheck<br/>buffer]
    ES --> SER
    SER -->|Series + Sketches| FWD[Forwarder]

The diagram is the same one drawn in pkg/aggregator/README.md, redrawn in Mermaid.

Key types

Type	File	Purpose
MetricSample	pkg/metrics/metric_sample.go	A single observation: name, value, type, tags, host, timestamp
Series	pkg/metrics/series.go	A flushed metric with one or more points
SketchSeries	pkg/metrics/sketch_series.go	A flushed distribution metric (DDSketch)
Event	pkg/metrics/event/	Free-form structured event
ServiceCheck	pkg/metrics/servicecheck/	OK / WARNING / CRITICAL / UNKNOWN signal
BufferedAggregator	pkg/aggregator/aggregator.go	The default in-memory aggregator
Demultiplexer	pkg/aggregator/demultiplexer.go	Routes samples between aggregators
TimeSampler	pkg/aggregator/time_sampler.go	DogStatsD aggregation by time bucket
CheckSampler	pkg/aggregator/check_sampler.go	Per-check-instance aggregation
ContextMetrics	pkg/aggregator/internal/...	Stores in-flight metrics keyed by their context
ContextResolver	pkg/aggregator/context_resolver.go	Maps (name, tags, host) to a stable context key

Sources of samples

• DogStatsD — every parsed StatsD line becomes a MetricSample and is pushed through the demultiplexer's DogStatsD path.
• Checks — every sender.Gauge(…) / Counter(…) / Histogram(…) call inside a check is converted to a MetricSample and pushed through the demultiplexer's checks path.

The two paths use different samplers because they have different semantics: DogStatsD samples come continuously and are bucketed by time, while check samples are batched at the end of each check run.

TimeSampler

TimeSampler is responsible for DogStatsD samples. It maintains a fixed-size time bucket (default 10 s) per metric context. When the bucket flushes, it emits a Series (or SketchSeries).

Implementation files:

• time_sampler.go — bucket logic.
• time_sampler_worker.go — the goroutine that drives the flush ticker.
• no_aggregation_stream_worker.go — bypasses sampling for "raw" pass-through mode.

DogStatsD also supports dogstatsd_no_aggregation_pipeline, which skips the sampler entirely and forwards points one-for-one. This is used for monitoring use cases where the operator needs the original cadence.

CheckSampler

Each check instance has its own CheckSampler. The check emits all its samples in one burst at the end of check.Run(), then commits. The CheckSampler aggregates within that single burst (so a check that emits the same gauge twice within a run only sends one point). It does not maintain longer-lived state — the check is expected to control its own cadence.

Demultiplexer

The demultiplexer (demultiplexer.go, plus per-flavor variants demultiplexer_agent.go, demultiplexer_serverless.go, demultiplexer_mock.go) is the dispatch hub. It owns the TimeSampler workers, routes samples to per-check CheckSamplers, and exposes the Sender API to checks.

Different flavors:

• Agent — full dispatcher with sampler workers and serializer.
• Serverless — minimal dispatcher tuned for short-lived processes.
• Mock — for tests; collects samples without aggregating.

Context resolution

Internally, the aggregator stores metrics keyed by context rather than by name + tags. A context is a small struct with a stable identifier; converting (name, tags, host) to a context happens once on receive and is cached in tagfilter_cache.go.

This is what makes hot DogStatsD paths fast: tag canonicalization and hashing happen at receive time, not at aggregation time.

Sketches and percentiles

Histograms and timings are aggregated as sketches rather than as raw values. Datadog uses DDSketch for relative-error-bounded percentile estimates. The aggregator emits SketchSeries payloads alongside Series; the backend accepts both.

The set of sketch percentiles emitted is configurable (histogram_aggregates, histogram_percentiles).

Events and service checks

The aggregator also routes events and service checks. They are not sampled; they are buffered and flushed periodically. The serializer encodes them with their own payload types.

Output

After flushing, the aggregator hands Series, SketchSeries, Event lists, and ServiceCheck lists to the serializer (pkg/serializer/ and comp/serializer/). The serializer produces wire-format payloads — JSON, MessagePack, or protobuf depending on payload type — and the forwarder (comp/forwarder/) ships them to the intake.

See Forwarder and serializer.

Configuration

Key	Effect
flush_interval	TimeSampler flush cadence (default 15 s on host, 10 s for some flavors)
histogram_aggregates, histogram_percentiles	Per-histogram aggregation
dogstatsd_no_aggregation_pipeline	Skip TimeSampler
aggregator_buffer_size	Internal channel buffer sizes
forwarder_apikey_validation_interval	(Forwarder) API key revalidation cadence

The aggregator's defaults are set in pkg/config/setup/config.go.

Testing

pkg/aggregator/test_common.go provides a MockSender that captures sender calls, and several helpers that pump time forward deterministically. New tests should use these helpers rather than time.Sleep.

The aggregator package has very heavy unit and benchmark coverage — see aggregator_test.go, time_sampler_test.go, check_sampler_bench_test.go, etc.

Entry points for modification

• New metric type: extend pkg/metrics/, add the corresponding aggregation logic in time_sampler.go and/or check_sampler.go.
• New aggregator flavor: implement a new Demultiplexer variant. Existing variants are good starting points.
• Tag filtering rules: edit tagfilter_cache.go and the taggerlist debug surface.

Related pages

• Apps: DogStatsD — the principal source of metrics.
• Systems: Check runtime — the other principal source.
• Systems: Forwarder and serializer — what comes after the aggregator.
• Features: Metrics pipeline — end-to-end view.