Ingest pipelines

What they are

Ingest pipelines are chains of processors that transform a document before it is indexed. They are configured via cluster-state metadata and run on any node with the ingest role.

Source layout

server/src/main/java/org/elasticsearch/ingest/
├── IngestService.java            Master-side pipeline registry; per-node executor
├── Pipeline.java, Processor.java Core abstractions
├── IngestDocument.java           Mutable doc passed through a pipeline
├── ConfigurationUtils.java       Common parsing helpers
└── ...

modules/ingest-common/            Built-in processors (set, rename, grok, gsub, append, ...)
modules/ingest-geoip/             GeoIP enrichment
modules/ingest-attachment/        Tika-based content extraction
modules/ingest-otel/              OTLP-compatible normalization
plugins/                          (none — ingest extensions live in modules)
x-pack/plugin/enrich/             enrich processor (lookup against an enrich index)
x-pack/plugin/redact/             redact processor (PII suppression via grok)
x-pack/plugin/geoip-enterprise-downloader/   Manages MaxMind databases

Lifecycle

graph LR
  Bulk[TransportBulkAction] --> IS[IngestService.executePipeline]
  IS --> P0[Processor 0]
  P0 --> P1[Processor 1]
  P1 --> P2[Processor 2]
  P2 -->|ok| OK[Bulk continues to shards]
  P2 -->|on_failure| OF[On-failure pipeline]

TransportBulkAction checks each indexing request for a default or per-request pipeline. If a pipeline applies, the request is detoured through IngestService.executePipeline on a node with the ingest role; the resulting document replaces the original and continues to the routing/shard step.

Built-in processors (sample)

Module	Processor	Purpose
ingest-common	set, rename, remove, script, gsub, lowercase, uppercase, split, join	Standard field manipulation
ingest-common	grok, dissect	Pattern-based parsing
ingest-common	convert, date, date_index_name	Type / time handling
ingest-common	pipeline	Call another pipeline
ingest-common	foreach	Apply processors to array elements
ingest-common	network_direction	Compute traffic direction from IPs
ingest-geoip	geoip	MaxMind enrichment
ingest-attachment	attachment	Extract text from binary documents (Tika)
enrich	enrich	Lookup against a reference index
redact	redact	Replace PII matches
ingest-otel	various	Normalize OTLP traces / logs

Conditional execution and on-failure

Each processor may carry an if script (Painless) and an on_failure block. The pipeline framework evaluates the condition and routes errors to the on-failure pipeline when present, otherwise marks the document as failed.

Persistent processors

Pipelines and processors that need long-lived state (like enrich, which builds a Lucene-backed lookup index, or geoip, which downloads MaxMind databases) hook into PersistentTaskPlugin or run scheduled tasks via the master.

Performance

Pipeline processors run synchronously on the ingest thread pool. Heavy processors (like Tika in attachment or large grok patterns) can dominate ingest latency; the project has failure_store and dedicated ingest nodes as common deployment patterns.

Where to extend

• New processor: implement Processor and Processor.Factory, register via IngestPlugin#getProcessors.
• New conditional language: not really — use Painless via the if field on each processor.
• Pipeline composition: use the pipeline processor to call other pipelines; this is the recommended way to share logic.