Telemetry, monitoring, and health

Purpose

The telemetry, monitoring, and health systems give operators a way to see what a cluster is doing: counters and gauges, distributed traces, slow logs, JVM diagnostics, and a structured "health report" that explains why a cluster is in the state it is.

Directory layout

server/src/main/java/org/elasticsearch/telemetry/
├── TelemetryProvider.java          Abstract provider (metrics + tracing)
├── metric/MeterRegistry.java       OpenTelemetry-style counters/gauges/histograms
├── tracing/Tracer.java             Span API
└── ...

server/src/main/java/org/elasticsearch/health/
├── HealthService.java              Cluster-wide diagnostic engine
├── HealthIndicatorService.java     Per-domain indicator interface
├── node/                           Per-node local health
└── ...

server/src/main/java/org/elasticsearch/monitor/
├── jvm/                            JvmInfo, JvmStats, HotThreads
├── fs/                             FsHealthService (disk health)
├── os/                             OS info
├── metrics/                        Indices/Node/SystemMetrics
└── process/

modules/apm/                       OpenTelemetry tracing + metrics shipper
x-pack/plugin/monitoring/          Legacy stack monitoring
x-pack/plugin/profiling/           Continuous profiling

Key abstractions

Type	File	Role
TelemetryProvider	server/src/main/java/org/elasticsearch/telemetry/TelemetryProvider.java	Pluggable metrics + tracing
MeterRegistry	.../telemetry/metric/MeterRegistry.java	OpenTelemetry-flavored API
Tracer	.../telemetry/tracing/Tracer.java	Span API
HealthService	server/src/main/java/org/elasticsearch/health/HealthService.java	Aggregates per-domain HealthIndicatorServices
HotThreads	server/src/main/java/org/elasticsearch/monitor/jvm/HotThreads.java	Sampling stack profiler

Stats endpoints

The classic counters live behind:

• GET /_nodes/stats — per-node JVM, OS, process, indices, transport, HTTP, ingest, thread pools.
• GET /_cluster/stats — cluster-wide rollup.
• GET /_cat/... — terse text views for ops dashboards.

Each subsystem registers its counters with IndicesMetrics, NodeMetrics, or its own MBean. Plugins contribute via the meter registry.

Tracing (modules/apm)

The apm module wires the Tracer SPI to OpenTelemetry exporters. With tracing.apm.enabled: true and an OTLP endpoint configured, each top-level transport action becomes a span; sub-spans capture shard-level work, allocator decisions, and snapshot uploads. Sampling and OTLP options are documented in TRACING.md.

Slow logs

Per-shard slow logs (server/src/main/java/org/elasticsearch/index/IndexingSlowLog.java, SearchSlowLog.java) record indexing and search operations exceeding configurable thresholds. Output goes to <cluster>_index_*_slowlog.log. Thresholds are dynamic per-index settings.

Health framework

GET /_health_report aggregates a tree of HealthIndicatorResult objects, each from a HealthIndicatorService that owns one diagnostic concern: master stability, shards availability, repository integrity, disk usage, ILM policy execution, etc. The framework attaches diagnoses with structured messages and remediation hints, so the response is consumable by automation, not just humans.

Indicators come from:

• Server: health-shards-availability, master stability, disk, repository integrity.
• X-Pack: ILM, SLM, ML.
• Modules: health-shards-availability, stateless-health-shards-availability.

Hot threads

GET /_nodes/hot_threads samples each thread's stack a few times over a configurable interval and returns the top consumers. Implemented in HotThreads, this is the project's first-look performance-debugging tool.

Disk-health monitoring

FsHealthService (server/src/main/java/org/elasticsearch/monitor/fs/FsHealthService.java) periodically writes and reads a tiny file in each data path; if the operation fails or is too slow it removes the node from the master's view, preventing it from sticking around with broken storage.

Continuous profiling

x-pack/plugin/profiling/ runs an eBPF-based collector that uploads stack samples to an Elasticsearch index, then provides flamegraph-style aggregations over them. It is the platform's "always-on profiler" for production diagnostics.

Stack monitoring (legacy)

x-pack/plugin/monitoring/ periodically writes node and cluster stats to a monitoring index (or remote cluster). This is the legacy "Stack Monitoring" feature — superseded by the OTLP-based apm module but still supported.

Entry points for modification

• New metric? Get the MeterRegistry from the TelemetryProvider and create a counter / histogram.
• New trace span? Use Tracer.startTrace(...), ensure the span is closed in a try/finally.
• New health indicator? Implement HealthIndicatorService, register via HealthPlugin (or the regular Plugin#createComponents returning the indicator).
• Replacing the telemetry backend? Implement TelemetryProvider and contribute via TelemetryPlugin.