Indexing

The path that takes a document from a REST _bulk request to a durable Lucene segment.

Entry points

• Single document: PUT/POST /<index>/_doc/<id> → RestIndexAction → TransportBulkAction (single-doc requests are routed through bulk for uniformity).
• Bulk: POST /_bulk → RestBulkAction → TransportBulkAction.
• Update: POST /<index>/_update/<id> → RestUpdateAction → TransportUpdateAction (a primary-side read-modify-write that falls back to bulk underneath).
• Delete by query / Update by query / Reindex: live in modules/reindex/.

High-level flow

graph TD
  C[Client] -->|HTTP| RB[RestBulkAction]
  RB --> TB[TransportBulkAction]
  TB --> IP[IngestPipelineService]
  TB --> AS[Auto-create indices via master]
  TB --> R[Routing: shard buckets]
  R --> TBSH[TransportShardBulkAction per primary]
  TBSH --> EN[InternalEngine.index/delete]
  EN --> LU[Lucene IndexWriter]
  EN --> TL[Translog append]
  TBSH --> RP[Replication to replica shards]
  RP --> ENR[Replica InternalEngine]

TransportBulkAction (server/src/main/java/org/elasticsearch/action/bulk/TransportBulkAction.java):

1. Optionally runs the request through ingest pipelines (resolved by index mapping or explicit ?pipeline=).
2. Auto-creates indices via master if they don't exist (subject to action.auto_create_index).
3. Splits the bulk request into per-shard buckets using the routing hash.
4. Forwards each bucket to the primary node via TransportShardBulkAction.
5. Aggregates per-shard responses into the final bulk response.

On the primary shard

IndexShard.applyIndexOperationOnPrimary (server/src/main/java/org/elasticsearch/index/shard/IndexShard.java):

1. Acquires a primary permit (cluster state may have changed).
2. Resolves the version (external version, sequence number, or document id).
3. Parses the source against the mapping (DocumentParser); if the mapping needs to be updated dynamically, the primary forwards the new mapping to the master and waits.
4. Calls InternalEngine.index():
   • Acquires a sequence number from LocalCheckpointTracker.
   • Writes to Lucene IndexWriter.
   • Appends a Translog.Operation.
5. Replicates the operation envelope to replicas.
6. Replicas re-execute deterministically using the seq-no/primary-term envelope; no parsing needed.

Backpressure

IndexingPressure (server/src/main/java/org/elasticsearch/index/IndexingPressure.java) tracks bytes-in-flight per node:

• Coordinating bytes — bytes received by the coordinating node before forwarding.
• Primary bytes — bytes accepted on the primary while indexing.
• Replica bytes — bytes received on replicas.

When a node exceeds indexing_pressure.memory.limit (default 10% of heap) it returns EsRejectedExecutionException and the bulk client must back off. Status is exposed via _nodes/stats?metric=indexing_pressure.

Mappings: dynamic vs strict

Mappings can be dynamic: true (auto-add new fields), false (ignore), strict (reject), runtime (add as runtime fields). The dynamic-update path goes through the master and may briefly block the indexing thread; high-throughput workloads should use explicit mappings or templates.

Refresh policies

A bulk request can carry refresh=true|wait_for|false:

• false (default): rely on the per-index refresh interval.
• wait_for: don't return until a refresh has covered the operation.
• true: force an immediate refresh (expensive; do not use under load).

Versioning and conflicts

• version_type=external / external_gt / external_gte use a user-supplied version.
• if_seq_no / if_primary_term use sequence-number-based optimistic concurrency.
• Internal: every operation gets a sequence number; conflicts surface as version_conflict_engine_exception.

Data streams

For time-series data, prefer a data stream (server/src/main/java/org/elasticsearch/cluster/metadata/DataStream.java). A data stream is a logical write target backed by a rolling sequence of hidden indices; rollovers are managed by ILM.

Where to extend

• Custom mappings / field types: MapperPlugin#getMappers.
• Pre-index transformation: ingest pipelines (IngestPlugin#getProcessors).
• Per-shard write listeners: IndexEventListener from Plugin#onIndexModule.
• Custom engine (e.g. searchable snapshot, stateless): EnginePlugin#getEngineFactory.
• Backpressure tuning: indexing_pressure.memory.limit and friends.