Remote Config

Purpose

Remote Config (RC) is the mechanism that lets Datadog push configuration overrides to running Agents from the backend. Customers don't have to redeploy the Agent to change settings; they update them in the Datadog UI, and the Agent picks them up over the next polling cycle.

RC powers a long list of features:

• APM samplers receive updated rates and rules.
• Autodiscovery receives new check templates.
• Cloud Workload Security receives new SECL policies.
• Logs pipeline receives source patches.
• Installer receives Agent upgrade and configuration commands.
• Database Monitoring receives query collection rules.
• APM Live Debugger receives probe definitions.

Roughly any feature with a "configurable from the UI" surface uses RC.

Layout

The Agent-side RC implementation lives in two places:

pkg/remoteconfig/         # Core RC client
└── state/, client/, ...

comp/remote-config/       # Component framework wrapper
├── rcclient/             # Client component
├── rcservice/            # Server component (Cluster Agent)
├── rctelemetryreporter/  # Self-telemetry
├── rcservicemrf/         # Multi-Region failover variant
└── rcstatus/             # Status integration

Architecture

graph LR
    BACKEND[Datadog backend<br/>RC service] -->|gRPC + signed payload| CLIENT[RC Client<br/>pkg/remoteconfig/client]
    CLIENT --> STATE[Local state store<br/>BoltDB]
    STATE -->|events| LISTENERS[Per-feature listeners<br/>APM, CWS, AD, ...]
    LISTENERS -->|apply config| FEATURE[Feature subsystem]
    DCA[Cluster Agent rcservice] -->|fan-out| NODES[Node Agents]

The client polls the backend over gRPC, validates payloads against bundled root certificates (pkg/remoteconfig/state/), and stores the latest state in a local BoltDB file. Per-feature listeners subscribe to changes and apply them in-process.

In Kubernetes deployments the Cluster Agent acts as a proxy: nodes don't poll the Datadog backend independently; they poll the Cluster Agent's rcservice which fans out a single backend connection to all nodes. This is what comp/remote-config/rcservice/ and rcservicemrf/ provide.

Trust model

Every RC payload is signed by Datadog's RC service. The Agent ships with the root public keys baked in (pkg/remoteconfig/state/); the verification is similar to TUF (the Update Framework). This means a hostile network or a compromised intermediary can't push bad configs.

Per-feature listeners

Subscribers register with the RC client at startup. Each listener gets the patches relevant to its product (APM, CWS, etc.). Examples in the codebase:

• pkg/trace/remoteconfighandler/ — applies sampler updates to the running Trace Agent.
• pkg/security/rconfig/ — applies SECL rule updates to CWS.
• comp/core/autodiscovery/providers/remote_config.go — turns RC payloads into autodiscovery templates.
• pkg/fleet/installer/rcaction.go — drives Installer actions.
• pkg/dyninst/ — receives dynamic instrumentation probes.

Configuration

Key	Effect
remote_configuration.enabled	Master switch (default true)
remote_configuration.refresh_interval	Polling cadence
remote_configuration.api_key	API key used for RC traffic
remote_configuration.config_root, director_root	Override the root keys (testing)
remote_configuration.directorjwt_*	JWT signing for the embedded fleet RC variant

Multi-Region Failover

comp/remote-config/rcservicemrf/ implements an MRF-aware RC service used by HA-deployed Cluster Agents. It watches multiple backends and fails over when the primary is unhealthy.

Inspection

agent remote-config status
agent remote-config dump-state

These subcommands (under cmd/agent/subcommands/remoteconfig/) print the local RC state, applied configurations, and the last update time per product.

Key abstractions

Type / package	Location	Description
Client	pkg/remoteconfig/client/client.go	gRPC poller and state machine
Repository	pkg/remoteconfig/state/repository.go	Local state store
Subscriber (per-product)	various	Receives applied configs
Service	comp/remote-config/rcservice/	Server-side fan-out (Cluster Agent)

Entry points for modification

• Adding a new product subscriber: register a callback with the RC client, parse the product's payload format, apply changes.
• Adding a new payload type to an existing product: extend the product-specific listener.
• New trust roots: rotate pkg/remoteconfig/state/ and update the bundled keys.

Related pages

• Apps: Cluster Agent — runs the RC fan-out in Kubernetes.
• Apps: Trace Agent — major RC consumer.
• Features: Cloud Workload Security — major RC consumer.
• Apps: Installer — RC-driven actions.