Indices and shards

Purpose

The indices and index packages own everything that happens once a request reaches the data layer: per-shard Lucene state, mapping resolution, the engine that drives writes and refreshes, the translog, and the per-index settings that govern the rest. A single Elasticsearch index is split into one or more shards; each shard is a complete Lucene index hosted on exactly one node at a time (with replicas elsewhere).

Directory layout

server/src/main/java/org/elasticsearch/indices/
├── IndicesService.java                Owner of all local IndexService instances
├── cluster/IndicesClusterStateService.java   Creates/deletes shards from cluster state
├── recovery/                          Peer recovery, retention leases, replica resync
├── analysis/                          Analyzer and tokenizer registry
├── breaker/                           Memory circuit breakers
├── store/                             Store + IndicesStore (manages shard data dirs)
├── flush/, refresh/                   Cluster-wide flush/refresh coordination
├── mapper/                            Field type registry helpers
└── ...

server/src/main/java/org/elasticsearch/index/
├── Index.java                         (name, UUID) tuple
├── IndexService.java                  Per-index components (mapping, similarity, analysis)
├── IndexSettings.java                 Per-index dynamic + static settings
├── IndexingPressure.java              Backpressure
├── shard/
│   ├── IndexShard.java                Per-shard lifecycle and operations
│   ├── ShardPath.java                 On-disk path
│   └── ...
├── engine/
│   ├── Engine.java                    Abstract engine
│   ├── InternalEngine.java            Default engine
│   ├── ReadOnlyEngine.java, FrozenEngine.java
│   └── ...
├── translog/                          Per-shard write-ahead log
├── mapper/                            FieldType, MappedFieldType, DocumentParser
├── codec/                             Custom Lucene codecs (incl. vector codecs)
├── search/                            Per-shard search builders, query rewriting
├── cache/                             Query cache, request cache, fielddata
└── ...

Key abstractions

Type	File	Role
IndicesService	server/src/main/java/org/elasticsearch/indices/IndicesService.java	Owns local indices; entry point for create / delete / lookup
IndexService	server/src/main/java/org/elasticsearch/index/IndexService.java	Per-index analyzers, similarity, mapping, shard map
IndexShard	server/src/main/java/org/elasticsearch/index/shard/IndexShard.java	The big one — operation entry point, lifecycle, recovery, replication
Engine	server/src/main/java/org/elasticsearch/index/engine/Engine.java	Lucene wrapper: index/delete/get/refresh/flush/forceMerge
InternalEngine	.../engine/InternalEngine.java	Default engine (Lucene IndexWriter + version map + translog)
Translog	server/src/main/java/org/elasticsearch/index/translog/Translog.java	Per-shard write-ahead log
MappedFieldType	server/src/main/java/org/elasticsearch/index/mapper/MappedFieldType.java	Polymorphic field representation used by queries
IndexingPressure	server/src/main/java/org/elasticsearch/index/IndexingPressure.java	Memory-based bulk backpressure
RecoverySource / RecoveryTarget	server/src/main/java/org/elasticsearch/indices/recovery/	Replica recovery state machine

Lifecycle of a shard

stateDiagram-v2
  [*] --> CREATED
  CREATED --> RECOVERING: assigned by allocator
  RECOVERING --> POST_RECOVERY: replica catches up
  POST_RECOVERY --> STARTED: master acknowledges
  STARTED --> CLOSED: index closed or relocating
  CLOSED --> [*]

IndexShard.startRecovery decides between store recovery (read from local Lucene + translog), peer recovery (copy from primary on another node), or snapshot recovery (restore from a repository). After recovery, the shard runs as primary or replica until reassigned.

Indexing path

sequenceDiagram
  participant C as Coordinating node
  participant P as Primary shard
  participant R as Replica shard
  C->>P: BulkShardRequest
  P->>P: parse, version-resolve, write to Lucene
  P->>P: append to translog
  P->>R: replicated request
  R->>R: write Lucene + translog
  R-->>P: ack
  P-->>C: BulkShardResponse

Documents are parsed by DocumentParser, validated against MappedFieldType, and handed to InternalEngine.index() which:

1. Resolves a sequence number (SeqNoStats) and primary term.
2. Optionally checks the version map for in-flight updates.
3. Writes to Lucene IndexWriter.
4. Appends a Translog.Operation to the translog.
5. Returns an Engine.IndexResult.

Replicas re-execute the operation deterministically using the seq-no/primary-term envelope from the primary.

Refresh, flush, merge

• Refresh: reopen the Lucene reader so newly indexed docs become searchable. Default 1s, configurable per index.
• Flush: commit the Lucene index and roll the translog. Triggered by translog size or explicit API.
• Force merge: merge segments down to a target number. Manual; not done automatically.
• Merge policy: MergePolicyConfig selects Lucene's TieredMergePolicy with project-specific tuning.

Mapping and field types

A document's mapping describes how fields are indexed and how queries against them rewrite. The mapper layer (server/src/main/java/org/elasticsearch/index/mapper/) compiles a JSON mapping into a tree of FieldMappers. Each FieldMapper produces a MappedFieldType used by query builders (MappedFieldType#termQuery, #rangeQuery, etc.). Custom field types are contributed by MapperPlugin.

Versioning and concurrency

Two version concepts coexist:

• External version — user-supplied or sequence-number-based optimistic concurrency, surfaced as _version.
• Sequence numbers / primary terms — internal causal-order metadata used for replica sync, retention leases, and CCR. See SequenceNumbers (server/src/main/java/org/elasticsearch/index/seqno/SequenceNumbers.java).

Caches and circuit breakers

• Request cache (indices.cache/) — caches whole shard-level query responses for cacheable searches.
• Query cache — Lucene-level filter cache with project-specific eviction.
• FieldData cache — for the few fields that still need columnar memory (text aggregations).
• Circuit breakers (indices.breaker/) — coarse memory ceilings (parent, fielddata, request, accounting). Trips throw CircuitBreakingException.

Integration points

• Receives shards via IndicesClusterStateService reading the routing table.
• Emits per-index events to IndexEventListener (used by ILM, transforms, ML, security audit).
• Backpressure feeds into IndexingPressureMonitor and ultimately into the bulk action's response.
• Engine plugins (e.g. searchable snapshots, frozen tier, stateless) provide alternative Engine implementations.

Entry points for modification

• New field type? Implement Mapper, MappedFieldType, parser; register via MapperPlugin#getMappers.
• New similarity (BM25 variant, etc.)? SimilarityService and SimilarityProvider.
• New engine? Extend Engine and contribute via EnginePlugin.
• New per-shard listener? IndexEventListener registered through Plugin#onIndexModule.
• Backpressure tuning? indexing_pressure.memory.limit and friends in IndexingPressure.