Cluster coordination

Active contributors: David Turner, Yang Wang, Henning Andersen

Purpose

Cluster coordination is the system that elects a master, replicates the cluster state document, and decides which node hosts which shard. It is the strongly-consistent backbone of Elasticsearch — every metadata change in the cluster (new index, mapping update, allocation decision, ILM transition, security role change) is applied as an atomic update to cluster state under a master.

Directory layout

server/src/main/java/org/elasticsearch/cluster/
├── ClusterState.java                       Immutable top-level state document
├── ClusterStateTaskExecutor.java           (currentState, tasks) -> newState contract
├── coordination/
│   ├── Coordinator.java                    Master election + publication state machine
│   ├── PreVoteCollector.java               Pre-voting (avoid term inflation under partitions)
│   ├── PublicationTransportHandler.java    Two-phase publish/commit of cluster state
│   ├── JoinHelper.java                     Node join workflow
│   ├── FollowersChecker.java / LeaderChecker.java   Bidirectional health checks
│   └── ...
├── service/
│   ├── ClusterService.java                 Public façade
│   ├── MasterService.java                  Serializes state-update tasks on the master
│   └── ClusterApplierService.java          Applies state on every node
├── routing/
│   ├── RoutingTable.java                   Per-shard routing
│   ├── allocation/
│   │   ├── AllocationService.java          Reroute entry point
│   │   ├── allocator/DesiredBalanceShardsAllocator.java   Modern allocator
│   │   ├── decider/                        AllocationDecider chain
│   │   └── ...
├── metadata/                               Index metadata, templates, aliases, system indices
├── node/DiscoveryNode.java, DiscoveryNodes.java      Membership snapshot
└── version/CompatibilityVersions.java      Per-node feature/transport-version negotiation

Key abstractions

Type	File	Role
ClusterState	server/src/main/java/org/elasticsearch/cluster/ClusterState.java	Immutable document containing Metadata, RoutingTable, DiscoveryNodes, blocks, customs
Coordinator	server/src/main/java/org/elasticsearch/cluster/coordination/Coordinator.java	Master-election + publication state machine
MasterService	server/src/main/java/org/elasticsearch/cluster/service/MasterService.java	Serializes update tasks on the master
ClusterApplierService	.../service/ClusterApplierService.java	Applies a published state on every node
AllocationService	.../routing/allocation/AllocationService.java	Computes the new RoutingTable
DesiredBalanceShardsAllocator	.../routing/allocation/allocator/DesiredBalanceShardsAllocator.java	Modern shard allocator
JoinHelper	.../coordination/JoinHelper.java	Adds nodes to the cluster
FollowersChecker / LeaderChecker	.../coordination/Followers*.java, .../Leader*.java	Health checks

How master election works

stateDiagram-v2
  [*] --> CANDIDATE
  CANDIDATE --> LEADER: pre-vote OK + join quorum
  CANDIDATE --> FOLLOWER: discovers a leader at higher term
  LEADER --> CANDIDATE: loss of quorum
  FOLLOWER --> CANDIDATE: leader check fails
  LEADER --> [*]

The Coordinator runs a Raft-inspired protocol:

1. On startup or quorum loss, every master-eligible node enters CANDIDATE mode.
2. Pre-voting (PreVoteCollector) probes peers without bumping the term, avoiding spurious election storms during partitions.
3. A candidate that sees a quorum of pre-vote responses requests joins from the voting configuration; if it collects a majority it becomes LEADER.
4. Followers are elected via JoinHelper and checked continuously by LeaderChecker; the leader checks them with FollowersChecker.
5. Loss of quorum kicks the leader back to candidate.

The voting configuration is stored in cluster state and changes only through carefully ordered reconfiguration steps.

Cluster state publication

State updates are produced on the master by MasterService and published in two phases:

sequenceDiagram
  participant Master
  participant F1 as Follower 1
  participant F2 as Follower 2
  Master->>F1: PUBLISH (state diff)
  Master->>F2: PUBLISH (state diff)
  F1-->>Master: ACK
  F2-->>Master: ACK
  Master->>Master: quorum reached
  Master->>F1: COMMIT
  Master->>F2: COMMIT
  F1-->>Master: COMMITTED
  F2-->>Master: COMMITTED

PublicationTransportHandler sends compressed cluster-state diffs (full state for new joiners). Followers apply through ClusterApplierService on a single thread.

Allocation

AllocationService.reroute() is invoked any time the cluster topology might require a re-evaluation: a node joins or leaves, an index is created or deleted, settings change, an ILM policy advances. The DesiredBalanceShardsAllocator computes a desired placement (a "balance"), then an executor moves shards toward that target without exceeding throttles.

The decision pipeline is a chain of AllocationDecider (server/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/), each capable of returning YES / NO / THROTTLE. Common deciders:

• AwarenessAllocationDecider — rack/zone awareness
• DiskThresholdDecider — refuses to allocate when disk would exceed thresholds
• ShardsLimitAllocationDecider — per-index and total shard caps per node
• FilterAllocationDecider — tag-based include/exclude
• NodeShutdownAllocationDecider — drains nodes that are shutting down

Persistence

Cluster state is persisted to disk by PersistedClusterStateService (server/src/main/java/org/elasticsearch/gateway/PersistedClusterStateService.java) and replayed on startup before the node joins. The on-disk format is a Lucene index (re-using the same engine for state durability).

Integration points

• Indices system consumes cluster state via IndicesClusterStateService to create/delete shards.
• Repositories and snapshots plug Custom documents into Metadata for persistent task tracking.
• X-Pack security registers role-mapping and license metadata as Custom entries.
• Plugins register ClusterStateTaskExecutors and Custom types via ClusterPlugin / Plugin#getNamedXContent.

Entry points for modification

• Adding a new metadata type? Implement Metadata.Custom and register it via ClusterPlugin.
• Adding an allocation rule? Implement AllocationDecider and register via ClusterPlugin#createAllocationDeciders.
• Tuning election timing? cluster.election.* settings in Coordinator.
• Touching publication? PublicationTransportHandler. Be very careful — wire format changes go through transport versions.