Notifier

Purpose

The notifier/ package dispatches alerts produced by alerting rules to external Alertmanager instances. It deduplicates, batches, and retries alerts; it discovers Alertmanagers via the same SD framework as scrape targets; and it emits per-Alertmanager metrics.

Directory layout

notifier/
├── manager.go            # notifier.Manager: queue, send loops, ApplyConfig.
├── alertmanager.go       # alertmanager type wrapping a single AM URL.
├── alertmanagerset.go    # alertmanagerSet: set of AMs sharing a config.
├── sendloop.go           # Independent send loops per AM (3.10+).
├── metric.go             # Self-metrics.
├── alert.go              # The Alert type (duplicated from common for stability).
├── util.go               # Helpers: dedupe, post.
└── *_test.go

Alert lifecycle

graph LR
    Rules[rules.Manager] -->|Send([]Alert)| Mgr[notifier.Manager]
    Mgr -->|enqueue| Q[per-AM-set queue]
    Q --> SL[alertmanagerSet.sendLoop]
    SL -->|HTTP POST /api/v2/alerts| AM[Alertmanager]
    AM -.-> SL

Manager.Send(alerts ...*Alert) is non-blocking and writes into per-set queues. Each alertmanagerSet runs its own send loop (sendloop.go), batches up to MaxBatchSize alerts (default 256), serialises to JSON, and POSTs to one Alertmanager from the set. Failures retry the next AM; prometheus_notifications_dropped_total{alertmanager} counts drops.

The "independent send loops per Alertmanager" change in 3.10 (#16355) means a slow Alertmanager no longer back-pressures sends to its peers.

Key types

Type	File	Role
Manager	notifier/manager.go	Top-level coordinator; owns the AM set map.
alertmanagerSet	notifier/alertmanagerset.go	Set of AMs sharing a config; one send loop per AM.
alertmanager	notifier/alertmanager.go	Wraps a single URL with HTTP client + relabeled URL.
Options	notifier/manager.go	Per-AM options (queue capacity, drain on shutdown, external labels).
Alert	notifier/alert.go	Send payload: labels, annotations, start/end-at, generator URL.

Discovery and relabeling

Alertmanager URLs are discovered via the same SD plugins as scrape targets: alertmanager_configs.<sd>_sd_configs. After SD, the manager applies alert_relabel_configs to filter or rewrite per-alert labels before sending.

Drain on shutdown

--alertmanager.drain-notification-queue-on-shutdown (default true) causes Manager.Stop to drain the queue before returning. Without this, alerts in flight at shutdown are lost.

Self-metrics

• prometheus_notifications_alertmanagers_discovered
• prometheus_notifications_dropped_total{alertmanager} — capacity overflow or send failure.
• prometheus_notifications_queue_capacity / _queue_length{alertmanager} (3.10+ added the AM dimension).
• prometheus_notifications_sent_total / _failed_total{alertmanager}.
• prometheus_notifications_latency_seconds.

The 3.10 PRs introduced the alertmanager label dimension on these metrics; older dashboards continue to work because the metric names did not change.

Configuration surface

prometheus.yml snippet:

alerting:
  alertmanagers:
    - timeout: 10s
      api_version: v2
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app]
          regex: alertmanager
          action: keep

CLI knobs:

• --alertmanager.notification-queue-capacity (default 10,000) — per-AM queue.
• --alertmanager.drain-notification-queue-on-shutdown.

Integration points

• rules.Manager calls Manager.Send on each iteration with the firing alerts.
• discovery.Manager feeds Alertmanager target groups.
• web API /api/v1/alertmanagers exposes the discovered + dropped AM URLs via web/api/v1.AlertmanagerRetriever.

Entry points for modification

• Add a new auth scheme: implement on the per-alertmanager HTTP client (notifier/alertmanager.go).
• Tune batch behaviour: the MaxBatchSize and queue logic live in sendloop.go.
• New alert payload field: changing Alert requires bumping the api version negotiation in alertmanagerset.go and a coordinated change in Alertmanager itself.

See Rules for where alerts originate.