Scrape

Active contributors: roidelapluie, bjboreham, krajorama, bwplotka

Purpose

The scrape/ package turns target groups produced by service discovery into samples appended to the storage. It owns one scrape pool per scrape_config and one scrape loop per target, runs HTTP requests on the scrape interval, parses the response, applies relabeling, and forwards everything to the Appender.

Directory layout

scrape/
├── manager.go                # scrape.Manager: top-level lifecycle + ApplyConfig.
├── scrape.go                 # scrapePool, scrapeLoop, scraper, target classification.
├── scrape_append_v2.go       # AppenderV2 path used when storage exposes the new shape.
├── target.go                 # Target struct, label set merging, hashing.
├── metrics.go                # All scrape self-metrics.
├── clientprotobuf.go         # Helpers for the application/protobuf scrape protocol.
├── helpers_test.go
├── manager_test.go
├── scrape_test.go            # ~6,900 lines of e2e scrape coverage.
├── target_test.go
└── testdata/                 # Fixture metric responses.

Lifecycle

graph LR
    Cfg[Config reload] --> M[scrape.Manager.ApplyConfig]
    SD[discovery.Manager] -->|target groups| M
    M --> Pool[scrapePool]
    Pool -->|per target| Loop[scrapeLoop]
    Loop -->|GET /metrics| Target[Target]
    Target --> Loop
    Loop -->|parse + relabel| App[storage.Appender]
    App --> Stor[(Storage / Agent WAL)]

The manager runs forever; pool reconciliation happens on every triggerReload signal. Adding or removing targets takes the per-pool targetMtx and updates the loop set without dropping samples for unchanged targets.

Key types

Type / Function	File	Role
Manager	scrape/manager.go	Top-level coordinator; owns the pool map.
scrapePool	scrape/scrape.go	Per-scrape_config set of loops; owns the HTTP client and symbol table.
scrapeLoop	scrape/scrape.go	Per-target ticker + scrape orchestration.
scraper	scrape/scrape.go	The HTTP client and parser instance.
Target	scrape/target.go	Discovered target with label set, schema, scheme.
targetActive/targetDropped	scrape/target.go	Filter sets exposed via /api/v1/targets.
Options	scrape/manager.go	Configuration container (HTTPClientOptions, FeatureRegistry, …).
FailureLogger	scrape/scrape.go	Captures bodies / errors when --scrape.failure-log-file is set.

Scrape protocols

Configurable via scrape_config.scrape_protocols. Supported protocols:

• PrometheusText0.0.4 — classic Prometheus text format.
• PrometheusProto — protobuf format with classic histograms.
• OpenMetricsText0.0.1, OpenMetricsText1.0.0.
• PrometheusText1.0.0 — UTF-8 metric/label names.

The parser is selected per-scrape based on the negotiated Content-Type header. Implementations live under model/textparse/.

Sample flow per scrape

1. scrapeLoop.scrape() makes the HTTP request with deadline = scrape_timeout.
2. The response body is decompressed (gzip/zstd) and handed to a Parser from model/textparse/.
3. For each line, Parser.Next() returns a (metric, timestamp, value, exemplar?, histogram?). The loop applies metric_relabel_configs to drop or rewrite samples.
4. Samples are batched into the current Appender. The cache (per-target string-interned symbol table) memoises labelsets to avoid re-allocating them every scrape.
5. On Commit(), the appender writes samples to the underlying storage.
6. The loop emits up, scrape_duration_seconds, scrape_samples_scraped, scrape_samples_post_metric_relabeling, scrape_series_added, and (when feature enabled) scrape_timeout_seconds, scrape_sample_limit, scrape_body_size_bytes.

Honour vs explicit timestamps

scrape_config.honor_timestamps controls whether timestamps in the response are kept. When false (the default for safety), the timestamp is reset to the loop's wall-clock; this avoids unbounded out-of-order ingestion when a target's clock drifts. The ScrapeTimestampTolerance (default 2 ms, scrape/scrape.go) further snaps timestamps onto a tolerance grid for better TSDB compression.

Sample limits and bounded retries

• sample_limit: drop and report failure if the scrape would exceed N post-relabel samples.
• body_size_limit: short-circuit on body size.
• label_limit, label_name_length_limit, label_value_length_limit.
• target_limit: per-pool cap on number of active targets.

A scrape that fails (HTTP error, timeout, parser error) marks the target down; up=0 is recorded for that scrape.

Symbol tables

Symbol tables (labels.SymbolTable) interned strings within a scrape loop. Periodically (lastSymbolTableCheck) the loop checks whether the table has grown too large versus its initial size and re-creates it. This is a memory regression guard and must remain — see #17472/#17500-era PRs.

AppendableV2 path

Scrape supports both storage.Appendable and storage.AppendableV2. With V2, exemplars and metadata travel with the sample, so the scrape loop avoids the second round-trip per sample. scrape/scrape_append_v2.go is the V2-specific coordinator.

The migration tracker is #17632.

Self-metrics

Prefix prometheus_target_* and prometheus_scrape_*:

• prometheus_target_interval_length_seconds — actual interval distribution.
• prometheus_target_scrape_pool_* — pool count, sync count, target count.
• prometheus_target_scrapes_* — scrapes per status code class.
• prometheus_target_scrape_pool_targets — current target count per pool.
• prometheus_target_sync_failed_total.
• prometheus_scrape_pool_target_limit (3.x).

Failure logging

When --scrape.failure-log-file is set, scrape.FailureLogger (a slog.Handler) writes JSON entries for failed scrapes (status code, error message, target labels). The log is rotated on SIGHUP.

Integration points

• Storage: receives an Appendable (or AppendableV2) from cmd/prometheus/main.go.
• Discovery: receives target groups from discovery.Manager over channels.
• Web API: /api/v1/targets, /api/v1/targets/metadata, and /api/v1/scrape_pools use the manager's TargetsActive/TargetsDropped/ScrapePools accessors via web/api/v1.TargetRetriever.
• Features: enables extra_scrape_metrics, start_timestamp_zero_ingestion, and type_and_unit_labels in the feature registry.

Entry points for modification

• New scrape protocol: add a parser to model/textparse/, register it in clientprotobuf.go::AcceptHeader, and update config/config.go validation.
• Tweak relabel pipeline: changes to model/relabel/relabel.go propagate everywhere the same package is imported (alerting too).
• Add a per-scrape config knob: edit config/config.go::ScrapeConfig, thread it through scrape.Options, and surface it to the loop.

See Discovery for where targets come from and Storage for where samples go.