System Probe

Active contributors: Bryce Kahle, Paul Cacheux, Guy Arbitman

Purpose

The System Probe is a privileged process that runs alongside the regular Agent and uses kernel-level mechanisms — primarily eBPF — to collect data the unprivileged Agent cannot. It exposes its data over a UNIX socket so the unprivileged core Agent and Security Agent can consume it.

System Probe powers:

• NPM (Network Performance Monitoring) — TCP/UDP flow tracking, conntrack, DNS resolution.
• USM (Universal Service Monitoring) — application-layer protocol decoding.
• eBPF-based core checks — TCP queue length, OOM kill, seccomp, etc.
• Kernel-level metrics for the Agent that need raw kernel access.
• GPU monitoring in pkg/gpu/.
• Discovery of services running on the host (pkg/discovery/).

Directory layout

cmd/system-probe/
├── main.go              # Linux entrypoint
├── main_windows.go      # Windows entrypoint (different mechanism)
├── command/             # Cobra root command
├── api/                 # HTTP server exposed over UDS
├── common/, subcommands/
├── modules/             # Each module is a self-contained probe
└── windows/

pkg/network/             # NPM data plane
├── ebpf/                # eBPF programs and Go bindings
├── tracer/              # The connection tracer
├── usm/                 # Universal Service Monitoring decoders
├── conn_filter.go, dns/, etc.

pkg/security/            # CWS data plane (also lives in system-probe address space)
pkg/ebpf/                # Shared eBPF infrastructure (loader, maps, BTF)
pkg/dyninst/             # Dynamic instrumentation
pkg/gpu/                 # GPU probes
pkg/discovery/           # Service discovery

bazel/rules/ebpf/        # Bazel macros for eBPF compilation

Modules

Each "module" is a self-contained subsystem that can be enabled/disabled independently in the system probe configuration. They live under cmd/system-probe/modules/. The most important:

Module	What it does
network_tracer	NPM core: TCP/UDP flow tracking, DNS resolution, conntrack.
universal_service_monitoring	USM: HTTP/HTTP2/gRPC/Postgres/MySQL/Kafka/Redis decoding.
ebpf	Generic eBPF probe management.
oom_kill, tcp_queue_length	Specific kernel checks.
language_detection	Process language identification.
runtime_security, compliance	CWS / CSPM in the system probe address space.
dynamic_instrumentation	dyninst-based dynamic tracing.
process	Process collection.
gpu	GPU telemetry collection.
discovery	Service discovery for the agent.
event_monitor	Generic kernel event firehose.
traceroute	Network path traceroutes.
pinger	ICMP/UDP reachability checks.
diagnose	Self-diagnostics.

The modules system, including registration and configuration plumbing, is in cmd/system-probe/modules/. There is detailed guidance in .cursor/rules/system_probe_modules.mdc for adding new modules.

eBPF infrastructure

pkg/ebpf/ is the shared infrastructure: program loader, map utilities, BTF helpers, runtime compilation support. Two convenience Bazel targets summarize the workflow:

# Build every eBPF .o program and runtime flattened .c file
bazel build //pkg/ebpf:all_ebpf_programs

# Verify all committed cgo godefs files are up to date
bazel test //pkg/ebpf:verify_generated_files

Programs are compiled in two ways:

• Pre-compiled at build time and shipped as .o files via Bazel's ebpf_prog/ebpf_program_suite rules.
• Runtime-compiled against the kernel's BTF at agent startup, via runtime_compilation_bundle. The Agent ships a flattened .c file and a clang invocation hash; the runtime compares the kernel's BTF and recompiles if needed.

For Co-RE-friendly programs, runtime compilation is unnecessary. Both styles coexist in the codebase.

For details, the root AGENTS.md "eBPF Bazel Build" section is the canonical reference.

NPM data flow

graph TB
    subgraph kernel[Linux kernel]
        EBPF[eBPF programs<br/>kprobes, tc, socket filters]
        MAPS[eBPF maps<br/>connections, ports, conntrack]
    end
    subgraph user[User space]
        TRACER[Tracer<br/>pkg/network/tracer]
        ENCODER[Encoder<br/>pkg/network/encoding]
        API[HTTP socket]
    end
    PROC[/proc, netlink] --> TRACER
    EBPF -. perf events .-> TRACER
    MAPS <-->|periodic dump| TRACER
    TRACER --> ENCODER
    ENCODER --> API
    API -->|UDS| AGENT[core Agent / Process Agent]

The network tracer (pkg/network/tracer/tracer.go) maintains the canonical view of active and recent connections. It pulls data from eBPF maps periodically, augments with conntrack/netlink, and exposes an HTTP endpoint (/network_tracer/connections) consumed by the Process Agent and core Agent.

Configuration

System Probe has its own configuration file (system-probe.yaml), separate from the main datadog.yaml. Each module is enabled with a flag like network_config.enabled: true or service_monitoring_config.enabled: true. Defaults live in pkg/config/setup/system_probe.go.

Subcommands

Subcommand	Purpose
run	The long-running privileged daemon (default).
version	Print version.
config	Read runtime config.
module-restart, modules	Inspect / restart modules.
runtime-security, compliance	CWS/CSPM helpers (mirrored on the security agent).

Windows

pkg/network/driver/ hosts the Windows kernel driver that replaces eBPF on Windows. The user-space surface is the same — same HTTP endpoints, same encoders — but the kernel side is a Windows Filtering Platform driver instead of eBPF. The driver's source lives in pkg/windowsdriver/.

Permissions and packaging

The System Probe runs as root on Linux and as LocalSystem on Windows. It is packaged separately from the main Agent in some distributions and signs its eBPF programs to satisfy the verifier. The systemd unit, packaging, and capabilities are in omnibus/, packages/, and cmd/system-probe/windows/.

Key abstractions

Type / package	Location	Purpose
Tracer	pkg/network/tracer/tracer.go	NPM connection tracker
EBPFTracer	pkg/network/tracer/connection/ebpf_tracer.go	eBPF-backed implementation
WindowsTracer	pkg/network/tracer/tracer_windows.go	Windows driver-backed implementation
Module	cmd/system-probe/api/module/module.go	Module abstraction
Manager	pkg/ebpf/manager.go	eBPF program/map manager
RuntimeCompilation	pkg/ebpf/bytecode/runtime/	Runtime-compiled program loader

Entry points for modification

• New eBPF probe: add the .c source under pkg/ebpf/c/ or a feature-specific directory, add a Bazel ebpf_prog target, and wire the loader in your module.
• New module: follow .cursor/rules/system_probe_modules.mdc and the existing modules under cmd/system-probe/modules/.
• New USM protocol decoder: extend pkg/network/usm/ with a new monitor.
• Cross-platform parity: pkg/network/tracer/ has separate _linux.go, _windows.go, _darwin.go files; new features almost always need attention on multiple platforms.

Related pages

• Features: eBPF / system probe — eBPF concepts for newcomers.
• Features: Network monitoring — NPM end-to-end.
• Features: Cloud Workload Security — CWS, which lives in the system probe address space.