Forwarder and serializer

Purpose

The forwarder is what gets data from the Agent to Datadog. The serializer is what turns in-memory payloads into the wire formats Datadog accepts. Together they handle authentication, retries, backoff, on-disk buffering, multi-region failover, and rate limiting.

These two systems used to live in pkg/forwarder/ and pkg/serializer/. Most of the logic now lives in comp/forwarder/ and comp/serializer/ (with pkg/serializer/ still hosting some older paths during the migration).

Architecture

graph TB
    AGG[Aggregator] -->|Series + Sketches + Events + Service Checks| SER[Serializer<br/>pkg/serializer]
    LOGS[Logs pipeline] -->|log payloads| EVPL[Event Platform Forwarder<br/>comp/forwarder/eventplatform]
    ORCH[Orchestrator] -->|orchestrator payloads| ORCHF[Orchestrator Forwarder<br/>comp/forwarder/orchestrator]
    SER -->|HTTP-ready bytes| FWD[Default Forwarder<br/>comp/forwarder/defaultforwarder]
    FWD --> RETRY[Retry queue<br/>memory + disk]
    FWD --> SEND[HTTP transactions]
    EVPL --> SEND
    ORCHF --> SEND
    SEND --> INTAKE[Datadog intake]
    SEND -. failure .-> RETRY
    RETRY -. retry .-> SEND

The serializer

pkg/serializer/ (with components in comp/serializer/) does payload encoding. Payload types include:

Payload	Format	Endpoint
Series	Protobuf (preferred) or JSON	/api/v2/series
Sketches	Protobuf	/api/beta/sketches
Events	JSON	/intake
Service checks	JSON	/api/v1/check_run
Host metadata	JSON	/intake

The serializer also handles:

• Compression — payloads are zstd-compressed if the configuration allows.
• Splitting — if a payload exceeds the intake's per-request size limit, the serializer splits it into multiple HTTP requests.
• Schema versioning — the serializer emits the right protobuf message version per Datadog account.

comp/serializer/metricscompression/ and comp/serializer/logscompression/ are the compression entry points.

The default forwarder

comp/forwarder/defaultforwarder/ is the canonical HTTP forwarder. Its main responsibilities:

• API key validation — periodically validates each configured API key against the intake.
• Multiple endpoints — supports primary and secondary endpoints (additional_endpoints in datadog.yaml) so customers can fan out to multiple Datadog accounts.
• Multi-Region Failover (MRF) — supports a parallel intake for HA setups; switches between regions on failure.
• Retry queue — failed transactions go onto an in-memory queue with exponential backoff. If the queue fills, transactions are evicted to disk under forwarder_storage_path (configurable; off by default).
• Rate limiting — concurrency caps per worker pool prevent the forwarder from overwhelming the intake.
• Worker pools — a fixed number of workers process the queue.

A recent fix (fix(debugger): drop proxy data when logs disabled, PR #49390) and [AGNTLOG-461] Fix connection_reset_interval not applied to additional/MRF endpoints (PR #49718) hint at the kinds of operational details the forwarder has to get right.

Disk-backed retry queue

When in-memory queueing isn't enough, the forwarder spills to disk. Configuration:

forwarder_storage_path: /var/lib/datadog-agent/forwarder
forwarder_storage_max_size_in_bytes: 5368709120 # 5 GB
forwarder_storage_max_disk_ratio: 0.95
forwarder_outdated_file_in_s: 86400 # drop after 24 hours

Disk-stored transactions are retried in order. Stale files are dropped on startup. Disk queueing is opt-in because it can cause unbounded disk usage during long network outages.

Specialized forwarders

Forwarder	Role
comp/forwarder/defaultforwarder	Series, sketches, events, service checks, host metadata.
comp/forwarder/eventplatform	Logs, CWS events, dyninst, container lifecycle, Database Monitoring payloads — anything that goes through Datadog's "Event Platform" endpoints.
comp/forwarder/orchestrator	Kubernetes orchestrator manifests.
comp/forwarder/connectionsforwarder	NPM connection payloads.
comp/forwarder/eventplatformreceiver	Receiver-side helper for the Event Platform protocol.

Each one is a thin wrapper around the same underlying HTTP client, configured for the right endpoint and payload type.

Authentication

The forwarder supports multiple API key configurations:

• Single API key — the default.
• Multiple keys per endpoint — when one customer wants payloads to fan out to multiple Datadog accounts.
• Secrets backend — keys can come from DD_API_KEY, the secret_backend_command, or any of the cloud provider secret stores.

API key validation runs periodically against the intake's /api/v1/validate endpoint and emits internal metrics on success/failure rates.

Telemetry

The forwarder is one of the most thoroughly self-instrumented subsystems in the Agent. It exposes (via expvar and agent telemetry):

• Transaction success / failure / dropped counts by endpoint.
• HTTP response code distribution.
• Per-endpoint latency.
• In-memory and on-disk queue sizes.
• API key validity by endpoint.

agent status includes a "Forwarder" section that surfaces all of this.

Key abstractions

Type / package	Location	Purpose
MarshalerProvider	pkg/serializer/marshaler.go	Pluggable payload marshalers
Serializer	pkg/serializer/serializer.go	Top-level serializer interface
Forwarder (interface)	comp/forwarder/defaultforwarder/forwarder.go	The forwarder API
DefaultForwarder	comp/forwarder/defaultforwarder/default_forwarder.go	The production implementation
Transaction	comp/forwarder/defaultforwarder/transaction/	Pending HTTP request abstraction
Resolver	comp/forwarder/defaultforwarder/resolver/	API key + endpoint resolver
EventPlatformForwarder	comp/forwarder/eventplatform/	Event Platform variant

Configuration

Key	Effect
dd_url	Primary intake URL
additional_endpoints	Map of extra endpoints to API keys
mrf_dd_url	Multi-region failover URL
forwarder_num_workers	Worker pool size
forwarder_timeout	Per-request timeout
forwarder_retry_queue_payloads_max_size	In-memory retry queue limit
forwarder_storage_path, forwarder_storage_max_size_in_bytes, forwarder_outdated_file_in_s	Disk-backed queue
connection_reset_interval	Periodically reset HTTP connections to refresh DNS

Entry points for modification

• New endpoint type: subclass the default forwarder or write a new wrapper in comp/forwarder/<name>/.
• New retry policy: extend comp/forwarder/defaultforwarder/transaction/.
• New compression algorithm: extend comp/serializer/metricscompression/ or comp/serializer/logscompression/.

Related pages

• Systems: Aggregator pipeline — what feeds the serializer.
• Features: Logs pipeline — uses the Event Platform forwarder.
• Reference: Configuration.