Python checks

Purpose

Datadog's integration ecosystem — hundreds of check implementations covering databases, message brokers, web servers, container runtimes, and more — is largely written in Python. The Agent embeds a CPython interpreter so those Python checks can run inside the same process as the Go-written agent core.

This is the part of the Agent that pulls together Go, C, C++, and Python in one compiled binary.

Layout

rtloader/
├── CMakeLists.txt              # CMake build (rtloader is a C++ library)
├── README.md
├── include/                    # rtloader public C ABI
├── rtloader/                   # rtloader implementation
├── two/                        # CPython 2 binding (Agent v6 only)
├── three/                      # CPython 3 binding (Agent v7+)
├── common/, cmake/, doxygen/, test/

pkg/collector/python/           # Go side of the Python check loader
├── api.go                      # Sender API (called from Python)
├── check.go                    # Python check abstraction
├── init.go                     # Interpreter setup
├── loader.go                   # Loader implementation
└── ...

cmd/agent/dist/checks/          # Bundled Python checks

How it works

graph LR
    subgraph go[Go side]
        LOADER[Python check loader<br/>pkg/collector/python]
        SENDER[Sender API]
    end
    subgraph cgo[CGO bridge]
        RTL[rtloader<br/>C++ library]
    end
    subgraph python[Python interpreter]
        CHECK[AgentCheck class<br/>Python integration]
    end
    LOADER -->|cgo calls| RTL
    RTL -->|Python C API| CHECK
    CHECK -->|metric / event / service_check| RTL
    RTL -->|cgo callbacks| SENDER
    SENDER --> AGG[Aggregator]

The Go agent invokes rtloader via cgo to instantiate the embedded interpreter and load Python check classes. When a check runs, Python code calls the AgentCheck class methods (e.g., gauge, counter, histogram); those methods are wired through rtloader and back into Go's Sender API.

Why a C++ shim?

Three reasons:

1. Isolation. The CPython API is large, version-specific, and not stable across major versions. The C++ shim presents a small, stable C ABI to Go (callable via cgo), insulating the Go side from CPython differences.
2. Version dispatch. The same rtloader interface has two implementations: two/ for CPython 2, three/ for CPython 3. Go calls the one selected by build tags.
3. Lifetime management. CPython's GIL and reference counting are easier to handle in C++ than via cgo function calls.

Two Python versions

Agent v6 still supports both Python 2 and Python 3 for backwards compatibility with checks that haven't been ported. Agent v7 dropped Python 2.

The two interpreters are mutually exclusive within a binary: a given build picks one based on build tags. Both code paths are compiled into the same Go module, but only one is linked into the rtloader instance.

Python integrations

Bundled Python checks live in integrations-core, a separate repository. The Agent ships them as wheels installed into its embedded site-packages. At startup the Python loader scans the site-packages for AgentCheck subclasses and registers them.

The bundled checks shipped with the Agent are also visible in cmd/agent/dist/checks/.

AgentCheck

Every Python integration extends the AgentCheck base class (provided by the datadog_checks_base package). The class exposes the same surface as the Go core check:

class MyCheck(AgentCheck):
    def check(self, instance):
        self.gauge("my.metric", 1.0, tags=["a:b"])
        self.service_check("my.check", AgentCheck.OK)

Calls into self.gauge, self.service_check, etc. resolve through rtloader to the Go-side Sender, where they enter the same aggregator pipeline as everything else.

Subprocess Python: the integration helper

agent integration (subcommand at cmd/agent/subcommands/integrations/) wraps pip install/uninstall against the Agent's embedded Python so operators can install third-party integration wheels without leaving the Agent's environment. It also enforces version compatibility constraints.

agent integration install -t datadog-myintegration==1.2.3
agent integration uninstall datadog-myintegration
agent integration freeze

Build

dda inv rtloader.make builds the C++ shim. dda inv agent.build triggers it transitively. Removing the build artifacts:

dda inv rtloader.clean

The Agent's CMake configuration lives at rtloader/CMakeLists.txt. CMake errors during build are common when the embedded Python version doesn't match the one on the host; cleaning rtloader and reinstalling tools usually resolves it.

JMX bridge

JMX checks are a special case. Instead of running Python, they bridge to a jmxfetch Java subprocess (pkg/jmxfetch/). The Python bridge is used to bootstrap each JMX check; the actual JMX work happens out-of-process.

Configuration discovery

Python checks consume the same autodiscovery configurations as Go core checks. The autodiscovery framework doesn't know whether the loader is Python or Go; it just hands the config to whichever loader recognizes it.

Internal observability

• Per-check stats include the language (Go / Python / JMX).
• The Python interpreter exposes its memory usage via agent status and via pkg/collector/python/.
• Errors in Python checks are reported as check-loading failures and surface in agent configcheck and agent check <name>.

Key abstractions

Type / package	Location	Description
Loader	pkg/collector/python/loader.go	Python check loader
PythonCheck	pkg/collector/python/check.go	Per-instance Python check
rtloader_t	rtloader/rtloader/RtLoader.cpp	C++ shim type
AgentCheck	datadog_checks_base (external)	Python base class

Entry points for modification

• New Python integration: contribute to integrations-core.
• New rtloader callback: extend the C ABI in rtloader/include/, add the C++ implementation in rtloader/two/ and rtloader/three/, and the Go binding in pkg/collector/python/.
• New Python version support: implement the binding under a new directory (e.g., four/) and add it to the build.

Related pages

• Systems: Check runtime — the unified check scheduler.
• Apps: Agent — the binary that hosts rtloader.