Machine learning

What it is

X-Pack ML is a suite of model-training and model-inference services that run inside the Elasticsearch cluster. It covers three product surfaces:

• Anomaly detection — unsupervised time-series anomaly detection (the original ML feature, going back to Prelert).
• Data frame analytics — supervised learning on tabular data (regression, classification, outlier detection).
• Trained model inference — serve PyTorch / Hugging Face models inside Elasticsearch, used by semantic_text, ELSER, multilingual e5, etc.

Source layout

x-pack/plugin/ml/
├── src/main/java/org/elasticsearch/xpack/ml/
│   ├── MachineLearning.java                  Plugin entry point
│   ├── job/                                  Anomaly detection job lifecycle
│   ├── datafeed/                             Datafeed lifecycle (pulls docs from indices)
│   ├── process/                              Bridge to native autodetect / categorize / data_frame_analyzer processes
│   ├── inference/                            Trained model loader + inference
│   ├── dataframe/                            Data frame analytics
│   ├── action/                               REST + transport actions
│   ├── notifications/                        Audit + alerts
│   └── ...
├── src/main/resources/                       Build descriptors for the native binaries
├── qa/                                       Integration tests
└── ...

x-pack/plugin/ml-package-loader/              Downloads and registers ELSER / e5 / etc.
x-pack/plugin/inference/                      Generic inference framework + service registry

Anomaly detection

Anomaly detection runs as a persistent task per job. The Java code orchestrates the job lifecycle (configuration, datafeed schedule, results indexing) and forks a native C++ autodetect process per job for the actual detection. The native process consumes JSON over stdin and emits results over stdout, which the Java side indexes into .ml-anomalies-* indices.

Datafeeds are little query-loops that pull batches of documents from a configured index pattern and stream them into the autodetect process. They run on the master-eligible-and-ml node that owns the job.

Data frame analytics

DFA jobs run as persistent tasks too, but the heavy lifting is in a native data_frame_analyzer C++ process. Java orchestrates training data extraction (via _search), forks the analyzer, captures the trained model, and stores it as a "trained model" in .ml-models.

Trained model inference

x-pack/plugin/ml/.../inference/ hosts a model-loading pipeline that supports:

• Native models (InferenceProcessor) — uses an embedded PyTorch via JNI for transformer models (text classification, embedding, NER, fill-mask).
• External services (x-pack/plugin/inference/) — calls hosted models at OpenAI, Hugging Face, Azure OpenAI, Cohere, Anthropic, Google Vertex, AlibabaCloud, etc.

Inference happens at index time (via the inference ingest processor) or at query time (via semantic_text and the inference action).

ELSER and the inference framework

x-pack/plugin/inference/ is a generalization of the older trained-model layer. It defines an InferenceService SPI plus a registry; each provider plugin (OpenAI, Cohere, ELSER) implements a service. The _inference API and the semantic_text field type both call into this layer.

ELSER (Elastic Learned Sparse EncodeR) is Elastic's own sparse retrieval model; ml-package-loader downloads it on demand and stores it as a trained model.

Native binaries

ML's native components live in a separate repository and are pulled in as binary dependencies during the Gradle build. The ml-cpp-snapshot task fetches them; the running node forks them as child processes when jobs start.

Where to extend

• New inference service (LLM provider): implement InferenceService + InferenceServiceFactory, register in InferencePlugin.
• New trained model type: extend InferenceConfig and the TrainedModelDefinition family.
• New data frame analytics analysis: extend the data_frame_analytics_config.analyses family — note this requires a matching change in the C++ analyzer.
• Tuning: xpack.ml.max_open_jobs, xpack.ml.max_machine_memory_percent, model-allocation auto-scaling settings.