Check runtime

Active contributors: Alexandre Yang, Nenad Noveljic, Guillermo Julián

Purpose

The Datadog Agent's primary collection mechanism is the check — a unit of code that runs on a schedule, queries something, and emits metrics, events, and service checks. Checks come in two flavors:

• Python checks — written in Python, loaded into the Agent via the embedded CPython interpreter (rtloader/).
• Core checks — written in Go, compiled directly into the binary (pkg/collector/corechecks/).

The collector schedules and runs both, dispatches metric samples through the same Sender API, and exposes per-check status in agent status.

Directory layout

pkg/collector/
├── README.md
├── check/                 # Check abstraction, configuration, scheduling primitives
├── corechecks/            # Go-implemented checks (CPU, memory, network, container, eBPF, …)
├── loaders/               # Loaders for different check sources
├── python/                # Python check loader (uses rtloader)
├── runner/                # Worker pool that runs scheduled checks
├── scheduler/             # Schedules checks at their configured intervals
├── infra_mode.go          # Infrastructure-mode (auto-enabled minimal checks)
├── stats.go               # Per-check stats
├── worker/                # Per-worker telemetry
└── sharedlibrary/         # Shared C library for Python checks

comp/collector/            # Component framework wrapping the collector
└── collector/             # Component definition

comp/checks/               # Component-based checks (Windows-specific, etc.)
├── agentcrashdetect/
├── windowseventlog/
└── winregistry/

Lifecycle

graph TB
    AD[Autodiscovery] -->|Config| SCHED[Scheduler<br/>pkg/collector/scheduler]
    LOADER[Loaders<br/>core / python / jmx] -->|Check instances| SCHED
    SCHED -->|Schedule by interval| RUNNER[Runner<br/>pkg/collector/runner]
    RUNNER --> CHECK[Check.Run]
    CHECK -->|MetricSample / Event / ServiceCheck| SENDER[Sender<br/>pkg/aggregator/sender]
    SENDER --> AGG[Aggregator]

Autodiscovery produces check configurations (a YAML blob plus identifying metadata). The collector hands each config to its loaders. The first loader that recognizes it returns a check instance; the scheduler then registers it for periodic execution. The runner is a worker pool that pulls from the scheduler and invokes check.Run() on each scheduled tick.

Loaders

pkg/collector/loaders/ defines the loader interface. Built-in loaders:

• Core check loader — instantiates Go checks registered via corechecks.RegisterCheck. Source: pkg/collector/corechecks/.
• Python check loader (pkg/collector/python/) — loads Python check classes from disk, instantiates them via the embedded interpreter.
• JMX check loader — bridges to the jmxfetch Java subprocess.

Loaders are queried in priority order; the first one that handles the config wins.

Core checks

Go-implemented checks live under pkg/collector/corechecks/:

Subdirectory	What it covers
system/	Host metrics (CPU, memory, disk, network)
containers/	Container runtime metrics
containerlifecycle/	Container start/stop events
containerimage/	Container image inventory
cluster/	Kubernetes cluster checks
kubernetes/	Kubernetes-specific checks
ebpf/	eBPF-driven kernel checks (TCP queue length, OOM kill, etc.)
gpu/	GPU usage
network/	NPM consumer check
networkdevice/	SNMP, NetFlow, network path
orchestrator/	Cluster Agent orchestrator data
oracle/, database/, dbms/	Database Monitoring checks
process/, host/, gohai/	Host inventory, process, host metadata

Each core check is a Go type that implements the Check interface from pkg/collector/check/. Adding a new core check follows the .claude/skills/create-core-check/ skill.

Python checks

pkg/collector/python/ and rtloader/ together form the Python check runtime:

• rtloader/ is a C++ library that embeds CPython. It exposes a small C ABI usable from Go via cgo. There are two builds — rtloader/two/ for Python 2 (Agent v6 only) and rtloader/three/ for Python 3 — selected by build tags.
• pkg/collector/python/ is the Go side: it instantiates the rtloader interpreter, finds check classes in installed Python packages, and routes calls through cgo.

Bundled checks ship under cmd/agent/dist/checks/. Operators can install third-party checks system-wide; the loader picks them up via Python's import mechanism.

Sender API

Inside a check, metrics are emitted via the sender:

sender, _ := check.GetSender()
sender.Gauge("my.metric", 1.0, "", []string{"key:value"})
sender.Commit()

The Sender abstraction lives in pkg/aggregator/sender/. It is provided by the aggregator component (comp/aggregator/). All metric/event/service-check emission goes through this single interface, so checks don't need to know about the aggregator's internals.

JMX checks

JMX-based checks bridge to the jmxfetch.jar Java subprocess. The check tells JMXFetch which beans to query; JMXFetch returns the values, which the Agent normalizes and sends to the aggregator. The bridge code is in pkg/jmxfetch/.

Cluster checks

Cluster checks are checks that should run once per cluster, not once per node. The Cluster Agent dispatches each cluster check to a registered Cluster Checks Runner (CLC), which is a slim core Agent with a check runner. See Apps: Cluster Agent for the dispatching side.

Status and introspection

• agent configcheck lists every loaded check configuration.
• agent check <name> runs a single check once and prints the resulting metrics/events/service checks (also useful for new check development).
• agent check <name> --rate runs the check twice with a delay so rate metrics produce real values.
• agent status shows per-check execution stats: last run timestamp, duration, success/failure.

Key abstractions

Type / package	Location	Description
Check (interface)	pkg/collector/check/check.go	What every check implements
Loader	pkg/collector/loaders/loader.go	Check-config-to-instance mapping
Scheduler	pkg/collector/scheduler/scheduler.go	Interval-based scheduling
Runner	pkg/collector/runner/runner.go	Worker pool
Sender	pkg/aggregator/sender/sender.go	Per-check sender API
Collector (component)	comp/collector/collector/	The component-framework wrapper

Entry points for modification

• Adding a core check: follow the .claude/skills/create-core-check/ skill. Implement Check, register via corechecks.RegisterCheck, add a default config under cmd/agent/dist/conf.d/.
• Adding a new Python integration: it lives in the integrations-core repository; the Agent picks it up at runtime if installed.
• Customizing scheduling: extend pkg/collector/scheduler/.
• Custom loaders are rare; existing core/python/jmx loaders cover most cases.

Related pages

• Systems: Autodiscovery — produces check configurations.
• Systems: Aggregator pipeline — what the Sender feeds into.
• Apps: Agent — the binary that hosts the check runtime.