Testing

The Datadog Agent has unit tests, integration tests, end-to-end tests, fuzz tests, and a kernel-matrix testing harness. They serve different purposes; running them all locally is rarely the right answer.

Unit tests

Most unit tests are Go tests that live next to the code as *_test.go. Run via Invoke, never raw go test:

# All unit tests in the repo
dda inv test

# A specific package
dda inv test --targets=./pkg/aggregator

# With coverage
dda inv test --targets=./pkg/aggregator --coverage

Why not go test? Because most files have build tags. Without the right combination the package won't even compile. dda inv test reads tasks/build_tags.py and applies the right tags per package.

You can also run tests through Bazel for packages that have a BUILD.bazel:

bazel test //pkg/aggregator/...
bazel test //comp/core/...

Python tests

Python checks and Invoke task helpers come with their own pytest suites:

dda inv invoke-unit-tests

Python lint:

dda inv linter.python

Mocking

The repo uses mockery, pinned via .mockery.yaml. Mocks are regenerated by dda inv … tasks (rather than by hand):

# Find the relevant generation task in tasks/<area>.py

Manually edited mocks are discouraged.

Integration tests

Integration tests sit in test/integration/ and test/gointegrationtest-related infrastructure. They run real backends (Docker containers, Kubernetes nodes via kind, etc.) inside CI and are exercised on the GitLab pipeline.

End-to-end tests

The E2E framework provisions cloud infrastructure, deploys the Agent, sends real workloads, and asserts on outgoing payloads via a mock intake. It is the safety net for changes that affect what reaches Datadog's intake.

Locations:

• test/new-e2e/tests/ — the test suites, organized by area (agent-platform, containers, npm, ot el, etc.).
• test/e2e-framework/ — the framework: Pulumi-based provisioners, agent params, environment helpers.
• test/fakeintake/ — a mock Datadog intake used by E2E tests to assert on outgoing payloads.

Run a single area:

dda inv new-e2e-tests.run --targets=./tests/agent-platform/...

By default fakeintake forwards captured payloads to the dddev Datadog org, so engineers can also inspect them in the real UI. See test/e2e-framework/AGENTS.md and test/fakeintake/AGENTS.md for the long-form documentation.

Critical: from the root AGENTS.md, "Most E2E tests only run on main, release branches (N.N.x), and RC tags — not on PR branches." Plan for that gap. Significant changes need the qa/rc-required label.

The repo also has a .claude/skills/write-e2e/ skill and a .claude/skills/run-e2e/ skill that AI assistants use to generate or run E2E tests.

Kernel Matrix Testing (KMT)

For eBPF and kernel-touching code, the project runs the same tests across many kernel versions in VMs:

# Build everything for KMT
dda inv kmt.gen-config
# See tasks/kmt.py and tasks/kernel_matrix_testing/ for the full surface

This is heavyweight; most contributors don't run it locally. CI runs the matrix on main and on PRs that touch eBPF code.

Documentation lives at tasks/kernel_matrix_testing/.

Fuzz tests

tasks/fuzz.py and tasks/fuzz_infra.py drive Go fuzzing for security-relevant code (notably SECL and trace agent payload parsing).

Test categories doc

The official taxonomy of test types lives in docs/public/guidelines/testing/test-categories.md. It distinguishes:

• Unit tests
• Integration tests
• E2E tests with the new framework
• Kernel matrix tests
• Fuzz tests
• Performance / benchmark tests in test/benchmarks/

Flaky tests

Known-flaky tests are tracked in flakes.yaml at the repo root. When a test is added there, CI ignores its failures. Removing entries from flakes.yaml is the project's "fix the flake" workflow — once the underlying bug is gone, the entry should go too.

Performance / regression tests

test/regression/ and test/benchmarks/ host benchmarks driven by Invoke (tasks/bench.py). They are run on dedicated infrastructure rather than from a developer machine.

Coverage

dda inv coverage and tasks/coverage.py generate coverage reports. The Codecov integration is configured in .codecov.yml. Coverage tooling lives in tasks/coverage.py.

How to write good tests for this repo

The patterns observed across the codebase:

• Use the component's Mock implementation. Most components in comp/<x>/<y>/mock/ provide a deterministic in-process mock. Wire it via Fx in tests.
• Avoid time.Sleep. The aggregator and forwarder packages have sophisticated test helpers that pump time forward deterministically — see pkg/aggregator/test_common.go.
• Cover the lifecycle. Tests for goroutine-spawning components should exercise Start() and Stop() paths. The most common production bugs come from send-on-closed-channel races at shutdown.
• Cover the platform you're not on. Run linters with GOOS=windows and GOOS=linux even if you only develop on macOS — easy way to catch build-tag mistakes.
• Validate fakeintake assertions in E2E tests. Asserting "the metric was emitted" is not the same as asserting "the right value reached the right intake endpoint."