envoyproxy/envoy

Stats

Envoy's stats system is one of its defining design choices: every line in the /stats admin output is a counter, gauge, or histogram updated lock-free from the data plane and aggregated periodically into a flush thread. The architecture is laid out in Matt Klein's stats blog post and the developer doc source/docs/stats.md. The implementation is in source/common/stats/.

Purpose

Make stat increments cheap on the hot path (atomic add, no locking).
Deduplicate stat names through a symbol table so that millions of cluster.foo.upstream_rq_2xx-style names cost almost no memory.
Expose the same stats via Prometheus, statsd, dog_statsd, OpenTelemetry, gRPC metrics service, hystrix dashboard, or the admin endpoint.
Allow scoping (per-cluster, per-listener, per-route, per-filter) without name explosion.

Layout

source/common/stats/
├── symbol_table.{h,cc}          # The string interning table
├── allocator_impl.{h,cc}        # Counter/gauge/histogram allocators
├── thread_local_store.{h,cc}    # Per-worker counter caches; merging into central store
├── histogram_impl.{h,cc}        # Per-stat histograms (HdrHistogram-based)
├── histogram_settings_impl.{h,cc}
├── store_impl.{h,cc}            # The root Stats::Store
├── tag_extractor_impl.{h,cc}    # Splits names like "cluster.x.upstream_rq" into tags
├── tag_producer_impl.{h,cc}     # Tag mode aware string composition
├── stats_matcher_impl.{h,cc}    # Allow/deny lists for stat name reporting
├── deferred_creation.h          # Lazy stat instantiation
└── isolated_store_impl.{h,cc}   # Test-only store

Key abstractions

Type	File	Purpose
`Stats::SymbolTable`	`symbol_table.h`	Interns dotted name fragments. A name like `cluster.x.upstream_rq_2xx` is a `StatName` of 4 symbol references.
`Stats::StatName`	(same)	Type-erased reference into the symbol table.
`Stats::ThreadLocalStoreImpl`	`thread_local_store.h`	The production store. Per-worker scopes hold per-worker counter caches; flush merges them.
`Stats::Scope`	`envoy/stats/scope.h`	A namespaced view (e.g. `cluster.x`) that creates counters under that prefix.
`Stats::Counter`, `Gauge`, `Histogram`	`envoy/stats/`	The metric types. Counter is monotonic; Gauge can rise and fall; Histogram is HdrHistogram-backed.
`Stats::Sink`	`envoy/stats/sink.h`	Pluggable export. Implementations under `source/extensions/stat_sinks/`.

The X-macro pattern

Every component declares its stats with a macro that the symbol table can compile down to a struct:

#define ALL_FOO_STATS(COUNTER, GAUGE, HISTOGRAM) \
  COUNTER(events) \
  COUNTER(errors) \
  GAUGE(in_flight, NeverImport) \
  HISTOGRAM(latency_ms, Milliseconds)

struct FooStats {
  ALL_FOO_STATS(GENERATE_COUNTER_STRUCT, GENERATE_GAUGE_STRUCT, GENERATE_HISTOGRAM_STRUCT)
};

FooStats stats{ALL_FOO_STATS(POOL_COUNTER_PREFIX(scope, "foo."),
                             POOL_GAUGE_PREFIX(scope, "foo."),
                             POOL_HISTOGRAM_PREFIX(scope, "foo."))};

The macros (defined in envoy/stats/stats_macros.h) generate a struct with one reference per metric. Every counter increment is a single non-atomic write to a thread-local + atomic add to the central counter at flush time.

The canonical example is at the top of source/server/server.h:

#define ALL_SERVER_STATS(COUNTER, GAUGE, HISTOGRAM) \
  COUNTER(debug_assertion_failures)                 \
  ...                                                \
  GAUGE(uptime, Accumulate)                          \
  HISTOGRAM(initialization_time_ms, Milliseconds)

Symbol table

A name like cluster.x.upstream_rq_2xx is split into four symbols (cluster, x, upstream_rq_2xx, plus the . separators are implicit). Each unique fragment is interned; the full name becomes a small array of symbol IDs. This is what makes ten thousand cluster metrics cost a few megabytes instead of hundreds.

Two implementations exist: a fake-symbol-table mode (for tests) and a real symbol table mode (production). The runtime feature flag selecting between them is being phased out — production has used the real table for years.

Per-worker caches

ThreadLocalStoreImpl keeps a per-worker cache of counters and gauges. A counter increment from a worker hits the worker-local cache without locks; the flush thread periodically aggregates worker caches into the central counter and sinks export the merged value.

Histograms have a different model: each worker has its own HdrHistogram; flush merges them.

Stat sinks

The configured sinks (source/extensions/stat_sinks/) run on the flush thread:

Sink	Path	Notes
`statsd` / `dog_statsd`	`statsd/`, `dog_statsd/`	UDP/TCP statsd; dog_statsd adds Datadog tag syntax.
`graphite_statsd`	`graphite_statsd/`	Graphite-flavoured statsd.
`metrics_service`	`metrics_service/`	gRPC streaming.
`open_telemetry`	`open_telemetry/`	OpenTelemetry export.
`hystrix`	`hystrix/`	Server-Sent Events for Hystrix dashboards.
`wasm`	`wasm/`	WASM plugin gets a stats stream.

The Prometheus endpoint is a handler on the admin server, not a sink — it pulls the current store on demand.

Stats matcher and inclusion lists

Operators can suppress unused stats with Bootstrap.stats_config.stats_matcher (stats_matcher_impl.cc). This is critical at scale: clusters with thousands of hosts can produce millions of stats; ignoring the unused ones reduces memory and flush time.

Tag extraction

TagExtractorImpl (tag_extractor_impl.cc) parses dotted stat names back into structured tags so that statsd-style sinks can emit (name, tag1=v1, tag2=v2) instead of the raw concatenated name. The default tag rules are in tag_producer_impl.cc and cover the well-known names (cluster, listener, route, virtual host, etc.).

Integration points

Server lifecycle. The store is constructed before everything else.
Workers. Each worker has thread-local caches.
Admin — /stats, /stats/prometheus, /stats/recentlookups, /clusters, /listeners — all read from the store.
Stat sinks — see above.
Flush thread — runs at stats_flush_interval (default 5s).

Entry points for modification

Adding a new metric to a subsystem: write the X-macro, generate the struct, instantiate from the appropriate scope.
Adding a new sink: implement Stats::Sink and register a Stats::SinkFactory.
Modifying tag extraction: edit tag_producer_impl.cc defaults or add a custom tag extractor in Bootstrap.stats_config.stats_tags.