Background

Notes on the why behind some of the choices visible in the codebase. This page collects design-decision context, known pitfalls, and migration history that doesn't fit neatly into the per-feature pages.

Why so many binaries?

The Agent ships as 24 separate binaries in cmd/. The reasons are operational and architectural:

• Privilege separation: the System Probe needs root for eBPF; the rest of the Agent should not. Splitting them into separate processes gives a clean privilege boundary.
• Performance isolation: the Trace Agent and Process Agent move large volumes of data and historically had different memory and CPU profiles than the metric-focused core. Separating them keeps one bad day from taking out the whole observability stack.
• Deployment flexibility: customers running container-only environments often deploy DogStatsD as a sidecar separately from the Agent. Cluster Agent runs once per cluster, not per node. Trace Agent can run standalone in serverless settings.
• Bundled-agent multiplexing: when the binaries do run on the same host, the agent binary (via the DD_BUNDLED_AGENT env var) can masquerade as several of the others, so they share an executable on disk while remaining logically distinct.

Why the component framework?

Before comp/, the Agent grew up using package-level globals: aggregator.GetDefaultSender(), config.Datadog, etc. That worked for a long time but made it nearly impossible to:

• Run two Agent instances in the same process (useful in tests).
• Wire a "minimal" Agent without dragging in the entire object graph.
• Mock subsystems for testing.

The Fx-based framework solves these problems. It also forces a stricter boundary between definitions (def/), implementations (impl/), and wiring (fx/), which makes the dependency graph explicit.

The migration is unfinished and likely will be for a long time. Most data-plane code (aggregator, trace agent, system probe) still lives in pkg/ and probably will, because the value of migrating it is small relative to the cost.

Why two build systems?

The Agent has been on Omnibus (Ruby) for years; the migration to Bazel started around 2023. Both produce real artifacts today.

Reasons for moving to Bazel:

• Faster builds: Bazel's content-addressed cache and per-target builds dramatically reduce CI times.
• Cross-language reproducibility: the Agent has Go, Python, C, C++, Rust, and eBPF C; Bazel handles all of them under one roof.
• eBPF: the eBPF build (compile per kernel/architecture combination, generate cgo godefs, produce runtime-compilation bundles) was painful with ad-hoc Make scripts; Bazel macros centralize it.
• Reproducible packaging: packages/ is a deterministic alternative to the Ruby-driven Omnibus packaging.

Why both still coexist:

• Migrating Omnibus completely is a large project. Some platform-specific packaging steps haven't been ported.
• Several legacy build tools (and the Agent's own release process) assume Omnibus is around.

You will see commits like "migrate another batch of packages to bazel" weekly. They are nibbling away at the migration.

Why is pkg/security/ so heavy?

Cloud Workload Security is the most active subsystem in the repo by lifetime commit volume. Several reasons:

• Continuous threat model evolution. New attacks, new evasion techniques, and new kernel features mean the rules and probes are perpetually being updated.
• Cross-kernel testing. CWS targets many kernel versions. Each kernel surface affects which probes work; even seemingly minor changes might require new code paths.
• Two products in one tree. CWS (runtime) and CSPM (compliance) share infrastructure but solve different problems.
• Performance-sensitive eBPF. Verifier limits, ring buffer pressure, and resolver caching are perennial concerns; the team revisits them often.

This is also why pkg/security/ and the security subsystems have so many sub-AGENTS.md files documenting conventions.

Pitfalls that have caused production bugs

From the root AGENTS.md review-guidelines section, paraphrased:

E2E coverage with fakeintake

The E2E framework asserts on outgoing payloads via fakeintake. Unit tests are not enough for changes that affect what reaches the intake. Many production regressions have been changes that the unit tests passed but that broke a payload format the backend cared about. Add an E2E test if your change touches metric serialization, log payloads, traces, or CWS events.

Branch-conditional CI

PR CI runs a subset of E2E. Most of the matrix only runs on main, release branches, and RC tags. This means classes of bugs — especially in packaging, startup/shutdown sequencing, and inter-process communication — won't be caught until after merge. The qa/rc-required label exists to flag changes that need extra QA scrutiny before the next RC.

Multi-platform divergence

The Agent ships on Linux, Windows, and macOS. Platform-specific code paths (build tags, OS-specific files, separate packaging) drift over time. The Linux deb/rpm and Windows MSI build paths are entirely independent. When making changes, look at the _windows.go siblings of your _linux.go files. Run GOOS=windows builds locally before merging.

Concurrency and lifecycle

Send-on-closed-channel races at component shutdown are the most-recurring concurrency bug. New goroutines need an explicit teardown story; new components must have tests that exercise both Start() and Stop().

Graceful degradation during startup

Components initialize in stages. Functions exposed to UIs and APIs (e.g., status, flare, expvar) must return safe defaults when their dependencies aren't ready, not panic or propagate errors. This is the source of "the agent crashes during startup" bugs.

Stale documentation

AGENTS.md and per-package documents are first-class artifacts. PRs that change behavior but leave docs untouched are flagged in review. AI agents are explicitly asked to update these files as they go.

Migration history snapshots

Historical context for big shifts visible in the tree:

• Agent v5 → v6 (2016): Python → Go rewrite. The reason this repo exists.
• Trace agent merge (2017–2018): the standalone trace-agent repo merged into this one.
• v7 + Python 3 (2019): drop Python 2 support.
• NPM and CWS (2020+): eBPF subsystems get serious investment.
• Component framework (2021–): comp/ introduced. Migration ongoing.
• Bazel (2023–): build system migration. Ongoing.
• OTel first-class (2023–): OTel collector flavor as a supported variant.
• AI assistants (2024–): AGENTS.md and .claude/skills/ become first-class.

Why the elaborate AGENTS.md ecosystem?

Datadog has been an early adopter of treating AI agent context as engineering documentation. The repo has:

• A root AGENTS.md with architecture, workflow, and review guidelines.

• Per-subtree AGENTS.md files refining repo-wide rules.

• .claude/skills/ defining task-specific procedures.

• An explicit norm in the root AGENTS.md:

  AI agents: when working on any task (reviewing, writing code, running tests), if you notice a gap or inaccuracy in an AGENTS.md or skill file, fix it.

The reasoning is straightforward: if AI assistants are going to operate in the codebase, the cost of inaccurate guidance compounds across many sessions. Updating AGENTS.md in the same PR as the code is the only way to keep that cost bounded.

Related pages

• Lore — the timeline view.
• How to contribute: Patterns and conventions — the rules these notes try to explain.
• How to contribute: Development workflow — process around the rules.