Autodiscovery

Purpose

Autodiscovery (AD) is the system that finds what the Agent should monitor on a host, then how. It watches container runtimes, Kubernetes APIs, the cluster filesystem, and other sources for services, matches them against templates (check configurations), and produces ready-to-schedule check configs that the collector picks up.

Without AD, every check would have to be statically configured. With AD, a customer can drop a redis: annotation on a Kubernetes pod and the Agent automatically starts running the Redis check against it.

Directory layout

The framework lives under comp/core/autodiscovery/:

comp/core/autodiscovery/
├── autodiscovery.go       # Top-level orchestration
├── component.go           # Fx component definition
├── config.go              # Internal config plumbing
├── configresolver/        # Resolves templates against discovered services
├── listeners/             # One implementation per discovery source
├── providers/             # One implementation per template source
├── scheduler/             # Subscriber abstraction
├── telemetry/             # AD self-telemetry
└── ...

Companion code in pkg/autodiscovery/ (legacy paths) and comp/core/workloadmeta/ (the canonical store of containers, pods, and ECS tasks).

How the pieces fit

graph LR
    subgraph sources[Discovery sources]
        DOCKER[Docker]
        KUBELET[Kubelet]
        ECS[ECS]
        CONTAINERD[containerd]
        STATIC[Static / file]
    end

    subgraph providers[Template sources]
        FILE[File templates<br/>conf.d/*.d/conf.yaml]
        ANN[Pod annotations<br/>labels]
        ETCD[etcd / Consul / ZK]
        REMOTE[Remote Config]
    end

    sources -->|listeners emit Service events| AD[Autodiscovery]
    providers -->|providers emit IntegrationConfig events| AD
    AD --> RES[ConfigResolver]
    RES -->|matched configs| SCHED[Scheduler subscribers]
    SCHED -->|configs| CHECKS[Collector check runtime]
    SCHED -->|configs| LOGSPIPE[Logs pipeline]
    SCHED -->|configs| TRACE[Trace receiver]

A listener observes a discovery source and emits Service events when something appears or disappears. A provider observes a template source and emits IntegrationConfig events. The config resolver correlates the two: when a Service matches a template's identifier, it produces a concrete check configuration with the service's host/port/tags filled in.

Subscribers — the collector, the logs agent, the trace receiver — register for config events. When they get one, they instantiate the corresponding action (start a check, start a log tailer, register a trace consumer).

Listeners

Each listener observes a different source. Notable implementations:

Listener	Source
Docker	Docker daemon socket
Kubelet	Kubelet API
containerd	containerd gRPC
ECS	ECS metadata API
Cloudfoundry	BBS / cf API
Environment	Static env-var-driven config
KubeServices	Kubernetes Services and Endpoints
KubeEndpoints	Cluster endpoint changes
SNMP	SNMP discovery
Remote Config	Templates pushed by the backend

Listeners produce Service objects with stable IDs, network info, and source labels. Workloadmeta is increasingly the source of truth: many listeners now subscribe to workloadmeta events rather than reaching out to the runtime themselves, which centralizes the cgroup/pid/container-id resolution.

Providers

Providers contribute templates. Notable ones:

Provider	Template source
File	conf.d/*.d/conf.yaml files on disk
Kubernetes pod annotations	ad.datadoghq.com/<container>.checks style annotations
Docker labels	com.datadoghq.ad.* labels
ECS task definitions	Task-level Datadog annotations
etcd / consul / zk	Distributed key-value template stores
Remote Config	Templates pushed dynamically by the backend
Cluster checks	Templates dispatched by the Cluster Agent
Container Lifecycle	Container start/stop events
Datastreams	Datastreams-specific config templates

ConfigResolver

comp/core/autodiscovery/configresolver/ does the matching. It evaluates template identifiers — typically image names, port numbers, label values — against the service's metadata. When it finds a match it substitutes template variables (%%host%%, %%port%%, %%pid%%) with concrete values and emits a resolved configuration.

Templates can match by:

• Image name (init_config: ad_identifiers: ["redis"]).
• Container name.
• Pod label.
• Custom identifier keyed off a label.

The resolver also implements template variables like %%host%%, %%port%%, %%pid%%, %%hostname%%, and lookup variables that read pod annotations or container env vars.

Subscribers

Anyone who wants to react to config events implements the scheduler.Scheduler interface from comp/core/autodiscovery/scheduler/. Subscribers include:

• The collector (for check configs).
• The logs agent (for log source configs).
• The trace agent (for trace ingestion configs in some integrations).
• Custom integrations.

Subscribers are notified on Schedule and Unschedule events; they are responsible for starting and stopping the corresponding work.

Cluster checks integration

Kubernetes cluster-level checks (one Kubernetes API check per cluster, not per node) are handled differently. The Cluster Agent runs its own AD providers, picks one node Agent (a Cluster Checks Runner) to handle each cluster check, and dispatches the resolved config to it. See Apps: Cluster Agent.

Key abstractions

Type / package	Location	Description
Service	comp/core/autodiscovery/listeners/types.go	Discovered runtime entity
IntegrationConfig	comp/core/autodiscovery/integration/config.go	Template config
Listener	comp/core/autodiscovery/listeners/types.go	Discovery source abstraction
Provider	comp/core/autodiscovery/providers/providers.go	Template source abstraction
ConfigResolver	comp/core/autodiscovery/configresolver/	Matches services to templates
Scheduler	comp/core/autodiscovery/scheduler/scheduler.go	Subscriber abstraction

Configuration

Key	Effect
confd_path	Where file-based templates live
additional_checksd	Where additional Python checks are loaded
autoconfig_template_dir_*	Distributed template store directories
autoconfig_from_environment	Use env-var-only static config
extra_listeners, listeners	Enable/disable specific listeners
extra_config_providers, config_providers	Enable/disable specific providers

Entry points for modification

• New listener: implement Listener, register it in bundle.go, and probably extend workloadmeta if the source produces container-like entities.
• New provider: implement Provider. The Remote Config provider is the most recent example.
• New template variable: extend configresolver/.
• A custom subscriber for a new integration type: implement Scheduler and register with the AD bundle.

Related pages

• Systems: Workloadmeta and tagger — the metadata store many listeners read from.
• Systems: Check runtime — the consumer of resolved check configs.
• Features: Logs pipeline — also a subscriber.