Inference and semantic search

What it is

The inference subsystem is the bridge between Elasticsearch and language models — both ones running inside the cluster (ELSER, multilingual e5, Hugging Face transformers via PyTorch JNI) and ones hosted externally (OpenAI, Cohere, Hugging Face Inference Endpoints, Anthropic, Google Vertex, Azure OpenAI, AlibabaCloud, Mistral, Watsonx, Voyage). Together with the semantic_text field type, it provides a "no-glue-code" semantic search flow.

Source layout

server/src/main/java/org/elasticsearch/inference/        Public SPI: InferenceService, ModelConfig
server/src/main/java/org/elasticsearch/index/mapper/vectors/SemanticTextFieldMapper.java

x-pack/plugin/inference/
├── src/main/java/org/elasticsearch/xpack/inference/
│   ├── InferencePlugin.java                  Registers REST + transport actions
│   ├── action/                               Inference actions
│   ├── services/                             One subpackage per provider
│   │   ├── elasticsearch/                    ELSER, e5, custom on-cluster models
│   │   ├── openai/, anthropic/, cohere/, googlevertexai/, azureopenai/, ...
│   ├── chunking/                             Strategies for splitting text before embedding
│   ├── registry/                             Endpoint registry stored in cluster state
│   ├── highlight/                            Semantic highlighting
│   └── ...
└── ...

x-pack/plugin/ml/.../inference/                Native (JNI) model inference for ELSER, e5, transformers
x-pack/plugin/ml-package-loader/               Downloads model artifacts on demand

Endpoints

PUT /_inference/text_embedding/<id>           Create / update an inference endpoint
GET /_inference/<task_type>/<id>              Read its config
POST /_inference/<task_type>/<id>             Run inference
DELETE /_inference/<task_type>/<id>

task_type is one of text_embedding, sparse_embedding, rerank, completion, chat_completion. Each provider declares which task types it supports.

The endpoint config includes the service (e.g. openai, elasticsearch), the underlying model id, credentials (stored in the keystore), and chunking settings.

Semantic_text

semantic_text is a field type that absorbs the boring parts of semantic search:

PUT my-index
{
  "mappings": {
    "properties": {
      "body": { "type": "semantic_text", "inference_id": "my-elser-endpoint" }
    }
  }
}

On ingest, SemanticTextFieldMapper chunks the text, calls the configured inference endpoint, and stores both the original source and the per-chunk embeddings in nested fields. Queries use the semantic query (or the semantic_text retriever) which routes to the right embedding type and assembles per-chunk hits.

SemanticTextFieldMapper.java lives in the server because it has to integrate with the indexing path; the actual model execution stays in the inference plugin.

Chunking

x-pack/plugin/inference/.../chunking/ provides several chunking strategies (word, sentence, none). Each chunk becomes a sub-document with its own embedding; queries can highlight at chunk granularity.

Highlighting

The semantic highlighter (registered by InferencePlugin) returns the chunks most relevant to the query, optionally scored against the embedding similarity.

Provider plumbing

Each provider in services/<name>/ implements:

• A request mapper (Elasticsearch model -> HTTP request body).
• A response parser.
• An HTTP client wrapping HttpAsyncClient.
• Rate limiting and retries.

Authentication credentials are stored in the keystore (SecureSetting) and never logged.

Inference processors and ingest pipelines

The inference ingest processor (x-pack/plugin/ml/.../inference/ingest/InferenceProcessor.java) runs a model against incoming documents, emitting a prediction or embedding into a target field. This is the older path; new code prefers semantic_text end to end.

Where to extend

• New external provider: add a services/<name>/ package, implement Service, ServiceFactory, Model, request/response types, and register in InferencePlugin.
• New on-cluster model format: modify services/elasticsearch/ and the underlying ML model loader.
• New chunking strategy: implement ChunkingService.Chunker and register.
• Tuning: per-endpoint rate limits and queue sizes; cluster-wide settings under xpack.inference.*.