Runners

Inference happens in a dedicated subprocess. The same ollama binary, when invoked as ollama runner …, dispatches to one of four backends. The daemon talks to each backend over a small HTTP API behind the llm.LlamaServer interface.

Purpose

Isolate inference from the daemon: a runner crash never takes down the API, model code can be swapped without restarting the daemon, and the same LlamaServer interface can wrap radically different runtimes (CGO into llama.cpp, pure Go, Apple MLX, image-generation).

Directory layout

runner/
├── runner.go            # dispatcher (4 lines of switch)
├── llamarunner/         # CGO wrapper around llama.cpp
├── ollamarunner/        # pure-Go runner
├── common/              # shared helpers
└── README.md

llm/
├── server.go            # LlamaServer interface + llmServer impl
├── status.go            # StatusWriter capturing runner stderr
└── llm_*.go             # platform-specific bits

llama/
├── llama.go             # CGO bindings into llama.cpp
├── llama.cpp/           # vendored upstream tree
├── patches/             # diff series applied on top
├── sampling_ext.cpp     # custom sampler glue
└── build-info.cpp       # generated build info

x/
├── mlxrunner/           # Apple MLX runner
└── imagegen/            # image-generation runner

Key abstractions

Symbol	Location	Purpose
runner.Execute	runner/runner.go	Top-level dispatcher: --ollama-engine → ollamarunner, --imagegen-engine → imagegen, --mlx-engine → mlxrunner, otherwise llamarunner.
LlamaServer interface	llm/server.go	What the daemon sees: Load, Ping, Completion, Embedding, Tokenize, Detokenize, Close, MemorySize, VRAMByGPU, ContextLength, etc.
llmServer	llm/server.go	Default implementation: spawns the runner subprocess, picks a free port, watches for exit, forwards HTTP calls.
StatusWriter	llm/status.go	Captures the runner's tail stderr so the daemon can include it in error responses.
llamarunner.Execute	runner/llamarunner/	CGO into llama.cpp; the historical default.
ollamarunner.Execute	runner/ollamarunner/	Pure-Go runner using ml/ and model/.
mlxrunner.Execute	x/mlxrunner/	Apple MLX runner.
imagegen.Execute	x/imagegen/	Image-generation runner.

How it works

graph TD
    Daemon[ollama serve] -->|spawn| Runner[ollama runner ...]
    Runner --> Dispatch[runner.Execute]
    Dispatch -->|default| Llama[llamarunner.Execute]
    Dispatch -->|--ollama-engine| Ollama[ollamarunner.Execute]
    Dispatch -->|--mlx-engine| MLX[mlxrunner.Execute]
    Dispatch -->|--imagegen-engine| Imagegen[imagegen.Execute]
    Llama -->|CGO| Cpp[llama.cpp]
    Ollama --> ML[ml/, model/, kvcache/, sample/]
    MLX --> MLXFW[Apple MLX framework]
    Imagegen --> ImagegenStack[mlx/, models/, safetensors]
    Llama -->|HTTP| Daemon
    Ollama -->|HTTP| Daemon
    MLX -->|HTTP| Daemon
    Imagegen -->|HTTP| Daemon

Each backend exposes the same minimal HTTP API (/health, /completion, /embedding, /tokenize, /detokenize). The daemon-side llmServer (in llm/server.go) is the single client the daemon ever talks to; the backends are interchangeable from the scheduler's point of view.

Choosing a runner

The default is llamarunner. The daemon picks an alternative when:

• The user / config sets OLLAMA_LLM_LIBRARY or OLLAMA_LLM_ENGINE to one of the alternatives.
• The model's metadata declares an engine (e.g., image-generation models force imagegen).
• The platform makes one preferable (Apple Silicon may prefer mlxrunner for some architectures).

scheduleRunner in server/routes.go explicitly deletes the deprecated use_imagegen_runner request option before scheduling — image generation is now driven by model metadata, not a per-request flag.

llamarunner (runner/llamarunner/)

The CGO bridge into llama.cpp lives in llama/llama.go. It exposes context creation, prompt evaluation, sampling, embedding, and tokenization. Custom samplers are linked in via llama/sampling_ext.cpp. The runner's HTTP server wraps these primitives in /completion, /embedding, and /tokenize routes.

Patches against the vendored llama.cpp tree are kept under llama/patches/ and reapplied through Makefile.sync when llama.cpp is updated.

ollamarunner (runner/ollamarunner/)

The pure-Go runner uses:

• ml/ for backend / device abstractions (ml/backend.go, ml/device.go).
• ml/backend/ for backend implementations.
• model/models/<arch>/ for per-architecture model logic (gemma, llama, qwen, mistral, deepseek, etc.).
• kvcache/ for KV cache implementations.
• sample/ for samplers.

Adding a new architecture means a new directory under model/models/ plus, usually, a converter under convert/ and a renderer/parser under model/renderers/ / model/parsers/ if the prompt format isn't standard.

mlxrunner (x/mlxrunner/)

A separate runtime targeting the MLX framework on Apple Silicon. Selected with --mlx-engine. Recent commits like mlxrunner: decouple models from attention cache storage layout and mlxrunner: apply RoPE at per-row positions show this is an actively-evolving backend.

imagegen (x/imagegen/)

The image-generation runner. Uses an MLX-based pipeline (x/imagegen/mlx/mlx.go is the largest file in the directory at ~2,400 lines). Endpoints /v1/images/generations and /v1/images/edits route through this backend.

Integration points

• The scheduler decides when to spawn and evict runners.
• The LLM subprocess wrapper (llm/server.go) is the single client interface.
• The model engine is what ollamarunner runs internally.
• Patches and sync targets are in tooling.

Entry points for modification

• New backend → add a runner.Execute switch case and implement the LlamaServer HTTP API.
• New architecture for ollamarunner → see model engine.
• Patch llama.cpp → drop a patch in llama/patches/ and run make -f Makefile.sync apply-patches.

Key source files

File	Purpose
runner/runner.go	Dispatcher.
llm/server.go	LlamaServer interface + llmServer impl.
llm/status.go	Tail-stderr capture for error reporting.
llama/llama.go	CGO bindings.
llama/sampling_ext.cpp	Custom samplers.
runner/llamarunner/	llama.cpp runner.
runner/ollamarunner/	Pure-Go runner.
x/mlxrunner/	MLX runner.
x/imagegen/	Image-generation runner.