eBPF / system probe

Purpose

eBPF is the foundational technology behind almost every "kernel-level" Datadog feature: NPM, USM, several core checks, and CWS. The Datadog Agent ships an eBPF infrastructure that handles loading, runtime compilation, BTF integration, and lifecycle management for tens of programs.

This page explains the eBPF infrastructure itself. For the products built on top, see Network monitoring and Cloud Workload Security.

Where eBPF lives

pkg/ebpf/                          # Shared infrastructure (loader, manager, helpers)
├── c/                             # eBPF C source (kernel side)
├── bytecode/runtime/              # Runtime-compiled bytecode bundles
├── manager.go, ringbuf*.go, ...

pkg/network/ebpf/                  # NPM eBPF programs and Go bindings
pkg/network/usm/ebpf/              # USM eBPF programs
pkg/security/ebpf/                 # CWS eBPF programs
pkg/collector/corechecks/ebpf/     # eBPF-driven core checks (TCP queue length, OOM kill, …)
pkg/dyninst/ebpf/                  # Dynamic instrumentation
pkg/gpu/ebpf/                      # GPU monitoring

bazel/rules/ebpf/                  # Bazel macros for eBPF compilation

Build pipeline

eBPF programs are built two ways:

Strategy	When	Where
Pre-compiled	At build time (in CI). The .o file is shipped alongside the Agent and loaded directly.	Most programs use this path. Bazel rule: ebpf_prog.
Runtime compilation	At Agent startup. The Agent ships a flattened .c source plus a clang invocation hash. The runtime compares the kernel's BTF and recompiles if needed.	Used for programs that need kernel-version-dependent constants. Bazel rule: runtime_compilation_bundle.

The convenience targets from AGENTS.md:

# Build every eBPF .o program and runtime flattened .c file at once
bazel build //pkg/ebpf:all_ebpf_programs

# Verify all committed cgo godefs files are up to date
bazel test //pkg/ebpf:verify_generated_files

# Update a single cgo godefs output
bazel run //pkg/ebpf:types_godefs

bazel/rules/ebpf/ defines the macros:

• ebpf_prog / ebpf_program_suite — compile .c → .o.
• cgo_godefs — go tool cgo -godefs + write_source_file verification.
• runtime_compilation_bundle — flatten headers + generate integrity hash .go file.

Loader

pkg/ebpf/manager.go wraps the upstream ebpf-manager library. It handles:

• Selecting between Co-RE, BTF, and runtime compilation based on what the kernel offers.
• Loading the right .o file for the running kernel.
• Pinning maps so programs can share state.
• Attaching probes (kprobes, uprobes, tracepoints, raw tracepoints, fentry/fexit, LSM hooks, tc, XDP, socket filters).
• Providing telemetry and verifier-friendly logs.

It also has fallback paths: if Co-RE-friendly programs fail to verify, the manager falls back to runtime-compiled equivalents.

Co-RE vs runtime compilation

• Co-RE (Compile Once, Run Everywhere) — the modern way. The program is compiled once with BPF type information; the kernel relocates field accesses at load time using its own BTF. No per-kernel build required.
• Runtime compilation — older path. The Agent ships C source, runs clang at startup against the kernel's headers/BTF, and produces a kernel-specific .o. Slower at startup but works on kernels that don't ship BTF.

Most new Datadog programs are Co-RE-friendly; some legacy ones still use runtime compilation.

BTF

BTF (BPF Type Format) is the metadata that lets Co-RE work. The Agent will:

1. Use the kernel's own BTF if available (/sys/kernel/btf/vmlinux).
2. Fall back to a bundled BTF tarball it ships for older kernels.
3. Fall back to runtime compilation if neither is available.

pkg/ebpf/btf.go and adjacent files implement the resolution.

Verifier-friendly programs

eBPF programs must pass the kernel verifier — a static analyzer that proves the program is bounded and safe. This is the main constraint on eBPF code.

Datadog's programs are written carefully: bounded loops, explicit memory bounds, no recursion, careful use of helper functions. Build tags and macros in pkg/ebpf/c/ parameterize over kernel versions to keep the verifier happy on a wide range of kernels.

Runtime checks

pkg/ebpf/uprobes/ handles user-space probes (e.g., for TLS-aware USM). pkg/ebpf/uprobes/usm/ provides USM-specific helpers.

pkg/network/protocols/ is the protocol-decoding layer used by USM kernel programs and their user-space monitors.

Kernel Matrix Testing

eBPF code is verified across many kernel versions in CI. The infrastructure lives in tasks/kmt.py and tasks/kernel_matrix_testing/. KMT spawns VMs at the right kernel versions, builds the Agent, and runs eBPF tests inside.

Locally:

dda inv kmt.gen-config
# See tasks/kmt.py for full options

KMT is heavyweight; the CI matrix runs it on PRs that touch pkg/ebpf/, pkg/network/, or pkg/security/.

Internal observability

• The Manager exposes verifier logs on load failure.
• expvar ebpf_* counters track program stats.
• system-probe diagnose runs a battery of self-checks.

eBPF-driven core checks

A class of core checks under pkg/collector/corechecks/ebpf/ run small eBPF programs to surface kernel-level signals as Datadog metrics. Examples:

• tcp_queue_length — TCP socket queue depths.
• oom_kill — OOM killer events.
• seccomp — seccomp denials.
• cwsm — CWS metrics.
• ksm — Kernel Same-page Merging.
• usm — USM telemetry.
• gpu — NVIDIA GPU usage.

Each check is split into:

• A kernel module under pkg/collector/corechecks/ebpf/probe/<check>/ that runs in the system probe.
• A check under pkg/collector/corechecks/ebpf/<check>/ that runs in the agent and calls the system probe over its socket.

pkg/collector/corechecks/ebpf/AGENTS.md documents the structure in detail.

Key abstractions

Type / package	Location	Description
Manager	pkg/ebpf/manager.go	The wrapped ebpf-manager
RingBuffer	pkg/ebpf/ringbuf*.go	Async event delivery from kernel to user
Telemetry	pkg/ebpf/telemetry/	eBPF program self-instrumentation
Bytecode	pkg/ebpf/bytecode/runtime/	Runtime-compiled program manifest

Entry points for modification

• New eBPF program: add .c source under the right feature directory, add a Bazel ebpf_prog target, write Go bindings, and load via the manager.
• New runtime-compilation bundle: use the runtime_compilation_bundle macro and add the program to pkg/ebpf/bytecode/.
• New eBPF-driven core check: follow pkg/collector/corechecks/ebpf/AGENTS.md.

Related pages

• Features: Network monitoring — biggest eBPF consumer.
• Features: Cloud Workload Security — second biggest.
• Apps: System Probe — the binary that runs all eBPF programs.