Discovery

Active contributors: roidelapluie, bjboreham, machine424

Purpose

Service discovery (SD) keeps the list of scrape and alertmanager targets up to date without hand-maintained lists. Each *_sd_config block in prometheus.yml becomes a Discoverer that emits targetgroup.Group updates over a channel. The discovery.Manager multiplexes them and pushes them to subscribers (the scrape manager and the notifier).

Directory layout

discovery/
├── README.md                 # Authoritative SD design doc.
├── discovery.go              # Discoverer, Config, Configs, StaticConfig.
├── manager.go                # discovery.Manager: ApplyConfig + multiplexing.
├── registry.go               # Reflect-based YAML codec for *_sd_configs.
├── refresh/                  # Helper for poll-based SDs (DNS, HTTP, …).
├── targetgroup/              # The Group type and JSON helpers.
├── install/                  # Empty package whose import side-effect registers all SDs.
├── metrics.go, metrics_*.go  # Per-mechanism metrics framework.
├── kubernetes/               # Endpoint, EndpointSlice, Pod, Node, Service, Ingress roles.
├── consul/  azure/  digitalocean/  dns/  eureka/  file/  gce/  hetzner/  http/
├── ionos/  linode/  marathon/  moby/  nomad/  openstack/  outscale/  ovhcloud/
├── puppetdb/  scaleway/  stackit/  triton/  uyuni/  vultr/  xds/  zookeeper/
└── aws/                      # EC2, ECS, ElastiCache, MSK, RDS, Lightsail roles.

How discovery configs become discoverers

graph LR
    YAML[prometheus.yml] -->|parse| Cfg[discovery.Configs<br/>list of Config]
    Cfg -->|RegisterConfig| Reg[type registry]
    Reg --> Mgr[discovery.Manager]
    Mgr -->|NewDiscoverer per Config| Disc[Discoverer]
    Disc -->|Run| Updates[targetgroup.Group channel]
    Updates --> Sub[subscribers:<br/>scrape.Manager,<br/>notifier.Manager]

discovery.RegisterConfig(...) is called from each SD package's init(). The Configs YAML codec uses reflection to wrap each list of configs as a typed entry under its registered name (registry.go::readConfigs).

The Discoverer interface

type Discoverer interface {
    Run(ctx context.Context, up chan<- []*targetgroup.Group)
}

Documented expectations (discovery/discovery.go, discovery/README.md):

• Send a full set of target groups initially.
• For updates, send the whole changed group (not deltas).
• Sources within an SD instance must be unique.
• Sending an empty Group{Targets: nil, Source: "x"} deletes that source.
• Don't close the channel.

Refresh helper

discovery/refresh/refresh.go wraps poll-based SDs (DNS, HTTP, OVH, etc.). Implementers provide a RefreshFunc and the helper handles the timer, retries, metrics, and channel mechanics.

Plugin registration

plugins/plugin_<name>.go files are tiny shims that import the SD package — and through that, trigger its init() to call discovery.RegisterConfig. Build tags select which SDs are compiled in:

• remove_all_sd — exclude every optional SD; keeps static_configs, file_sd, and http_sd.
• enable_<name>_sd — re-add a specific SD when remove_all_sd is set.

This was added in 3.10 (#17736) to let users build slimmer Prometheus binaries.

Built-in SDs

About 25 service discoveries live in this repo. Each implements Discoverer and emits its own __meta_<sd>_* labels. Notable ones:

SD	Notes
static_configs	Hard-coded targets directly in prometheus.yml.
file_sd_configs	Watches local YAML/JSON files for target groups; the canonical extension point.
http_sd_configs	Pulls target groups from an HTTP endpoint; documented in docs/http_sd.md.
kubernetes_sd_configs	Pod/Service/Endpoint/EndpointSlice/Node/Ingress; uses client-go.
consul_sd_configs	Health-API based with optional health_filter (3.11).
aws/ec2_sd_configs	EC2 instance discovery; AWS SDK v2.
aws/ecs, aws/elasticache, aws/kafka, aws/lightsail, aws/rds	Specialised AWS roles.
azure_sd_configs	Azure resource-manager VMs; supports Workload Identity (3.11).
dns_sd_configs	SRV / A / AAAA / MX / NS records.
digitalocean, linode, vultr, hetzner, gce, ionos, outscale, ovhcloud, scaleway, stackit	Cloud-vendor SDs.
marathon_sd_configs, nomad_sd_configs	Orchestrator SDs.
eureka_sd_configs	Netflix Eureka registry.
triton_sd_configs	SmartOS / Triton.
xds_sd_configs	Envoy/xDS discovery.
zookeeper_sd_configs	Pulls targets from Zookeeper paths.

The full list, with config field tables, is in docs/configuration/configuration.md (the per-SD sections under <scrape_config>).

Metrics

• prometheus_sd_received_updates_total / _updates_delayed_total / _updates_total
• prometheus_sd_discovered_targets
• prometheus_sd_kubernetes_*, prometheus_sd_consul_*, etc.
• prometheus_sd_last_update_timestamp_seconds (3.11) — when the last update was sent to consumers (#18194).

Mapping SD metadata to Prometheus

The general rule (from discovery/README.md):

• All metadata is exposed as __meta_<sdname>_<key> labels.
• Arrays become a single comma-prefixed-and-suffixed string (,a,b,c,) so relabel regex matches work.
• Maps become per-key labels, sanitised to [_a-zA-Z0-9].
• Multi-port targets use one target per port (Kubernetes) or named-port labels.

The user picks what they need with relabel_configs — there is no business logic in the SD itself.

Adding a new SD

discovery/README.md is the up-to-date checklist. Summary:

1. Implement Discoverer + Config in a new sub-package.
2. Register the config in an init() via discovery.RegisterConfig.
3. Add a plugins/plugin_<name>.go file with build tags remove_all_sd && enable_<name>_sd || !remove_all_sd.
4. Add a <name>_sd_configs entry to <scrape_config> and <alertmanager_config> documentation in docs/configuration/configuration.md.
5. Tag the maintainer in MAINTAINERS.md and CODEOWNERS.
6. Validate that a config containing the new SD survives DeepEqual round-tripping by adding it to config/testdata/conf.good.yml.

Entry points for modification

• Tweak the manager: discovery/manager.go — backoff, dedup, per-pool channels.
• Improve the refresh helper: discovery/refresh/refresh.go — used by ~10 SDs; performance changes propagate widely.
• New role for an existing SD: add the role to the SD's package, update its Config.Validate, and add a test under config/testdata/.

See Configuration for the YAML schema.