Logs pipeline

Active contributors: Alexandre Jacquemot, Brian Floersch, Dustin J. Mitchell

Purpose

The logs pipeline is how the Datadog Agent collects log lines from many sources, normalizes them, and ships them to the Datadog Logs intake. The bulk of the implementation lives in pkg/logs/, with the component-framework wrapper in comp/logs/.

End-to-end view

graph LR
    SRC1[Files / journald / Docker / kubernetes / TCP / UDP / Windows ETW] --> TAILER[Tailers]
    TAILER -->|Message| PIPE[Pipeline]
    PIPE --> PROC[Processors<br/>parsing, scrubbing, sampling]
    PROC --> BATCH[Batcher]
    BATCH --> SENDER[HTTP Sender]
    SENDER -->|/api/v2/logs| INTAKE[Datadog logs intake]

Sources

pkg/logs/sources/ defines the source abstraction. pkg/logs/launchers/ provides one launcher per source type:

Launcher	What it tails
file/	Plain log files. Discovers via globs and Kubernetes/Docker conventions.
journald/	systemd-journald. Linux-only.
docker/	Docker container stdout/stderr via the Docker daemon.
kubernetes/	Container logs collected via Kubernetes pod log files.
windowsevent/, windowsevents/	Windows Event Log via ETW.
tcp/, udp/	Network log receivers (syslog-like).
traps/	SNMP trap forwarding to logs.
eventplatform_receiver/	Internal mechanism for components that already produce logs payloads.
container/	Generic container runtime tailer (Docker, containerd, CRI).
service/	Generic service tailer driven by autodiscovery.
agentcrashdetect/ (Windows)	Crash dump files.

Sources are produced by autodiscovery the same way check configurations are. A pod annotation like ad.datadoghq.com/<container>.logs: '[{...}]' produces a logs source that the pipeline picks up.

Pipeline stages

pkg/logs/internal/ and the component bundle host the in-memory pipeline:

1. Tailer reads bytes from the source and emits Message records.
2. Decoder turns bytes into UTF-8 strings, handles multi-line and encoding detection.
3. Processor applies:
   • Parsing (e.g., docker JSON envelope, journald metadata).
   • Scrubbing — regex-based redaction of secrets and PII (pkg/logs/internal/processor/).
   • Adaptive sampling — recently extended to support per-source overrides ([AGNTLOG-587] Add per-source adaptive sampler config parity, PR #49955).
4. Batcher assembles messages into upload-size-friendly chunks.
5. HTTP sender ships the chunk through the Event Platform forwarder.

The pipeline is a chain of channels with backpressure; if the sender is slow, the tailers slow down naturally.

Logs vs. log events

Logs are sent as JSON arrays to /api/v2/logs. The wire format includes:

• message — the log line (after multi-line aggregation).
• service, source, host, tags — identifying metadata.
• timestamp — best-effort original timestamp.
• Other origin-specific metadata (docker.container_id, kubernetes.pod_name, …).

Tagging comes from the same tagger that the metrics pipeline uses. The container ID resolved by the launcher gets translated into the full set of pod/container tags transparently.

Configuration

Logs collection is opt-in:

logs_enabled: true
logs_config:
  container_collect_all: true # tail all container stdout/stderr
  use_compression: true
  compression_level: 6
  compression_kind: zstd
  batch_max_size: 1000
  batch_max_concurrent_send: 0 # unlimited
  open_files_limit: 200

Per-source configuration files go under conf.d/<name>.d/conf.yaml with a logs: array, or via Kubernetes annotations.

A recent fix worth highlighting (fix(logs/journald): first journal entry dropped when tailing from beginning, PR #48933) shows how subtle the per-source semantics can be.

Adaptive sampling

To control volume the pipeline supports adaptive sampling: it keeps a target rate per source by stochastically dropping messages when the source's rate exceeds the target. Configuration is per-source; recent work has extended parity across additional and MRF endpoints.

Multi-region failover for logs

Logs use the Event Platform forwarder which supports MRF. The same mrf_dd_url-style addressing applies; recent fixes around connection_reset_interval (PR #49718) explicitly covered the additional/MRF endpoint case for logs.

Logs library

comp/logs-library/ is a more recent addition: it carves out parts of the logs pipeline so they can be reused by binaries that don't want the full Agent (the OTel Agent in particular). Recent commits like [AGNTLOG-510] Add pkg/logs/processor to the logs-library (PR #45595) show this migration in progress.

Internal observability

• agent streamlogs — live stream of log events as they pass through the pipeline (one of the core Agent's subcommands).
• agent analyzelogs — analyze a captured logs payload offline.
• expvar logs.* counters and gauges.
• agent status Logs section.

Key abstractions

Type / package	Location	Description
Source	pkg/logs/sources/source.go	Logs source abstraction
Tailer	pkg/logs/tailers/	Per-source tailer interface
Launcher	pkg/logs/launchers/	Discovers sources and starts tailers
Message	pkg/logs/message/message.go	A single log event in flight
Pipeline	pkg/logs/pipeline/pipeline.go	The processing chain
Processor	pkg/logs/internal/processor/	Parsing, scrubbing, sampling
Auditor	pkg/logs/auditor/	Tracks tailer offsets so logs survive restarts
Sender	pkg/logs/sender/	HTTP sender

Entry points for modification

• New source: implement a Tailer and a Launcher under pkg/logs/launchers/<name>/ and pkg/logs/tailers/<name>/.
• New processor: extend pkg/logs/internal/processor/.
• New autodiscovery integration: extend comp/logs/adscheduler/.

Related pages

• Systems: Autodiscovery — produces logs sources from annotations.
• Systems: Forwarder and serializer — the Event Platform forwarder ships logs.
• Apps: Agent — the binary that hosts the logs pipeline.