Debugging

How to figure out what a running Elasticsearch process is doing.

Logs

Default logs go to logs/elasticsearch.log (or stdout when running under :run). Per-shard slow logs go to:

• logs/<cluster>_index_indexing_slowlog.log
• logs/<cluster>_index_search_slowlog.log

Configuration is in config/log4j2.properties (or distribution/src/config/log4j2.properties when running from source).

Turning up logging at runtime

PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.cluster.coordination": "TRACE",
    "logger.org.elasticsearch.indices.recovery": "DEBUG"
  }
}

The convention in the codebase is logger.<package>.<Class> keys. Package-level keys cascade.

Slow logs

server/src/main/java/org/elasticsearch/index/IndexingSlowLog.java and SearchSlowLog.java log per-shard operations whose duration crosses a configurable threshold:

PUT /my-index/_settings
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.indexing.slowlog.threshold.index.warn": "1s"
}

Diagnostic REST endpoints

A short menu of endpoints that should be the first reach when something is off:

Endpoint	What it tells you
GET /_cluster/health?wait_for_status=yellow&timeout=30s	Is the cluster reachable, what is its color, are there unassigned shards
GET /_cluster/allocation/explain	Why a particular shard is not where you expect
GET /_nodes/hot_threads?threads=10	What threads on each node are doing right now
GET /_nodes/stats?level=indices	Node-level + index-level counters
GET /_cat/pending_tasks	Backlog of master-state updates
GET /_cat/recovery?active_only	In-progress shard recoveries
GET /_cat/thread_pool?v	Thread pool occupancy and queue lengths
GET /_health_report	The newer health framework's structured report

Hot threads

hot_threads (server/src/main/java/org/elasticsearch/monitor/jvm/HotThreads.java) samples each thread's stack twice over a configurable interval and ranks them by CPU time. It is the project's "first port of call" for runtime-perf complaints.

Attaching a debugger

For tests:

./gradlew :server:test --tests <ClassName> --debug-jvm

For a running node:

./gradlew :run --debug-jvm

Both make the JVM listen on port 5005. From IntelliJ, create a "Remote JVM Debug" run configuration on localhost:5005.

Capturing heap and thread dumps

# Heap dump
jcmd <pid> GC.heap_dump /tmp/es.hprof

# Thread dump
jcmd <pid> Thread.print > /tmp/es-threads.txt

The data directory (config/jvm.options.d/) is configured to dump on OOM by default.

Diagnostic dump

The Elastic support team often asks for a support-diagnostics bundle. The relevant repo is elastic/support-diagnostics — it gathers cluster state, node stats, hot threads, allocation explanations, and logs into a single archive.

Common errors and where to look

Symptom	First places to look
MasterNotDiscoveredException	cluster.coordination logs, _cluster/allocation/explain, network reachability between master-eligible nodes
circuit_breaking_exception	indices.breaker.* settings, _nodes/stats?metric=breaker
Bulk rejections	indexing.write thread pool queue, IndexingPressure counters
Slow searches	search slow log, _nodes/hot_threads?type=search, profile=true on the query
Shard fails to allocate	_cluster/allocation/explain, gateway logs for missing shard data
Transport timeout	transport.connect_timeout, network partitioning, mismatched JVM versions
Translog recovery hangs	recovery logs, _cat/recovery?active_only

Tracing

TRACING.md in the repo root explains how to enable OpenTelemetry tracing via the apm module. Set tracing.apm.enabled: true in elasticsearch.yml and configure an OTLP endpoint to ship spans to your APM stack.

Allocator decisions

The cluster.routing.allocation family of loggers (especially DesiredBalanceShardsAllocator) explains why shards were placed where they were. Set them to DEBUG if the allocator is making surprising choices.