ES|QL (Elasticsearch Query Language)

Active contributors: Nik Everett, Andrei Stefan, Bogdan Pintea, Costin Leau

What it is

ES|QL is a piped, schema-on-read query language for Elasticsearch — a peer to KQL, Splunk SPL, and Kusto KQL. It compiles to a vectorized, columnar execution plan and runs against the same Lucene shards that back classic search. ES|QL is the most actively developed subsystem in the repository today.

FROM logs-*
| WHERE @timestamp > NOW() - 1h
| EVAL response_kb = response_bytes / 1024
| STATS p99 = PERCENTILE(response_kb, 99) BY service
| SORT p99 DESC
| LIMIT 10

Source layout

x-pack/plugin/esql/
├── src/main/java/org/elasticsearch/xpack/esql/
│   ├── action/                                  EsqlQueryAction, EsqlCapabilities
│   ├── parser/                                  ANTLR grammar (EsqlBase{Lexer,Parser}.g4)
│   ├── analysis/                                Analyzer, Verifier, type resolution
│   ├── optimizer/                               Logical & physical plan optimizers
│   ├── planner/                                 LocalExecutionPlanner, mapping plan -> operators
│   ├── plugin/                                  EsqlPlugin (entry point)
│   ├── session/                                 EsqlSession (one query lifecycle)
│   ├── expression/function/                     Function registry, scalar + agg + rank
│   ├── enrich/                                  Enrich policies + lookup
│   └── ...
├── compute/                                     The compute engine (page-block model)
│   ├── data/                                    Block, Page (columnar batches)
│   ├── operator/                                Source / filter / project / aggregator operators
│   ├── aggregation/                             Per-block aggregator implementations
│   └── lucene/                                  ValuesReader-driven Lucene source operators
├── qa/                                          CSV-spec tests, REST tests
└── ...

x-pack/plugin/esql-core/                         Shared types (DataType, Literal, ...)
x-pack/plugin/esql-datasource-*/                 Pluggable data sources (Parquet, S3, GCS, CSV, Iceberg, ...)

Query lifecycle

graph TD
  REST[REST: POST /_query] --> Parse[ANTLR parser]
  Parse --> AST[Logical plan AST]
  AST --> Analyze[Analyzer: bind names, types]
  Analyze --> Verify[Verifier: semantic checks]
  Verify --> LO[LogicalPlanOptimizer]
  LO --> Map[Mapper: logical -> physical plan]
  Map --> PO[PhysicalPlanOptimizer]
  PO --> LEP[LocalExecutionPlanner]
  LEP --> Driver[Driver: pipeline of operators]
  Driver --> Lucene[ValuesReader-driven Lucene source]
  Driver --> Result[EsqlQueryResponse]

Key files:

• EsqlQueryAction (x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlQueryAction.java) — public action.
• EsqlSession (.../session/EsqlSession.java) — orchestrates the lifecycle of one query.
• Analyzer (.../analysis/Analyzer.java) — type resolution and name binding.
• LogicalPlanOptimizer and LocalLogicalPlanOptimizer (.../optimizer/) — rule-based plan rewrites.
• LocalExecutionPlanner (.../planner/LocalExecutionPlanner.java) — turns a physical plan into operators on a Driver.

The compute engine

ES|QL execution is column-oriented:

• Block — a typed columnar buffer of values (int, long, double, BytesRef, etc.) plus null mask.
• Page — a tuple of blocks (one per column) sharing a row count.
• Operator — getOutput() / addInput() operators consume and produce pages.
• Driver — runs a pipeline of operators on a single shard or coordinator.

A typical pipeline might be: LuceneSourceOperator → EvalOperator → FilterOperator → HashAggregationOperator → OutputOperator. Operators are stateless across pages where possible; aggregators carry per-group state.

The compute engine is in x-pack/plugin/esql/compute/. It reuses Lucene IndexReader low-level APIs but never goes through MappedFieldType for hot-path reads — it asks for ValuesReaders directly, which is where most of the speed comes from.

Functions

EsqlFunctionRegistry (x-pack/plugin/esql/.../function/EsqlFunctionRegistry.java) is the registry of built-in functions: scalars (COALESCE, LENGTH, STARTS_WITH, math, date, IP, geo), aggregations (COUNT, SUM, AVG, PERCENTILE, VALUES, TOP), full-text (MATCH, KQL), spatial (ST_CONTAINS, ST_DISTANCE, ST_INTERSECTS), vector (KNN, V_*), and rank functions.

Adding a function: implement an EsqlScalarFunction / EsqlAggregateFunction, generate the operator/aggregator (often via templates and code generation in compute/), and register in EsqlFunctionRegistry.

Cross-cluster ES|QL

ES|QL queries can target remote clusters via <cluster>:<index>. The coordinator splits the plan into per-cluster sub-plans, ships them to the remotes, and merges the resulting pages.

Pluggable data sources

Beyond Lucene, ES|QL has a growing set of datasources under x-pack/plugin/esql-datasource-*/. These let FROM target Parquet files in S3, GCS, or Azure; CSV; Iceberg tables; and a few compression formats. Each datasource registers a ValuesReader-style adapter so the compute engine can run unchanged against external data.

Capabilities and version negotiation

EsqlCapabilities.java is a flat enum of every ES|QL feature. In a mixed-version cluster the coordinator advertises which capabilities it understands; the planner avoids generating physical plans that require a capability not present on every participating node. This is the busiest file in the ES|QL package and the single biggest source of recent commits.

CSV-spec tests

ES|QL has its own test format. Each *.csv-spec file under x-pack/plugin/esql/qa/testFixtures/src/main/resources/ contains a sequence of named blocks: a query, expected schema, expected rows. They run via CsvIT and are the project's preferred way to capture an ES|QL behavior:

stats_first_last
required_capability: agg_first_last
FROM employees
| STATS first = FIRST(salary BY emp_no), last = LAST(salary BY emp_no)
;

first:long | last:long
36174       | 74999

Run a single one:

./gradlew :x-pack:plugin:esql:internalClusterTest --tests "...CsvIT.*stats_first_last*"

Where to extend

• New function: register in EsqlFunctionRegistry, generate operator (compute/operator).
• New optimization rule: subclass LogicalPlanOptimizer.OptimizerRule (or its physical counterpart).
• New data source: implement under x-pack/plugin/esql-datasource-* and register via the EsqlPlugin SPI.
• New capability: add to EsqlCapabilities and reference from the planner where it matters.