Model engine

The pure-Go inference stack used by ollamarunner. It spans ml/, model/, kvcache/, sample/, and tokenizer/.

Purpose

Run inference for an Ollama model without depending on llama.cpp. Each architecture has a Go implementation that constructs a forward pass over a tensor backend, with KV cache, sampling, and tokenizer modules provided as separate packages.

Directory layout

ml/
├── backend.go             # SystemInfo, BackendMemory, Backend interface
├── device.go              # DeviceInfo, DeviceID, FilteredRunnerDiscovery
├── path.go
├── backend/               # backend implementations
└── nn/                    # neural-network primitives (linear, attention, layernorm, ...)

model/
├── model.go               # Model interface (Forward, Vocab, ...)
├── imageproc/             # image preprocessing for vision models
├── input/                 # token / image input types
├── models/                # one subdirectory per architecture
│   ├── gemma2/, gemma3/, gemma4/, llama/, llama4/, mistral/, qwen3/,
│   ├── deepseek2/, mllama/, gpt-oss/, ... (~20+ architectures)
├── parsers/               # parse model output back into Message
└── renderers/             # produce model-specific prompts

kvcache/
├── cache.go
├── causal.go              # causal KV cache
├── encoder.go             # encoder KV cache
├── wrapper.go             # composable wrappers
└── *_test.go

sample/
├── samplers.go            # temperature, top-k, top-p, mirostat, ...
└── *_test.go

tokenizer/
├── tokenizer.go
├── bpe/, sentencepiece/, ...
└── *_test.go

Key abstractions

Symbol	Location	Purpose
Backend	ml/backend.go	The tensor backend interface — context creation, tensor ops, scheduling.
Tensor, Context	ml/backend.go	The numerical primitives.
Model	model/model.go	Per-architecture model. Each model/models/<arch>/model.go implements this.
Cache	kvcache/cache.go	KV cache interface; Causal and Encoder are concrete implementations.
Sampler	sample/samplers.go	Decoding strategies.
Renderer	model/renderers/	Builds the model's specific prompt format.
Parser	model/parsers/	Inverse of renderer: parse output back into api.Messages.

How it works

graph TD
    Request[ChatRequest] --> Renderer[model/renderers]
    Renderer --> Tokens[input tokens]
    Tokens --> Forward[Model.Forward]
    Forward --> Backend[ml/backend]
    Backend --> KVCache[kvcache]
    Forward --> Logits[output logits]
    Logits --> Sampler[sample]
    Sampler --> Token[next token]
    Token --> Detokenize[tokenizer]
    Detokenize --> ParseStream[model/parsers]
    ParseStream --> Response[Message stream]

The runner loop in runner/ollamarunner/ wires these together: render prompt → feed prompt to model → sample → detokenize → parse → emit chunk.

Architectures

Each model architecture lives in its own subdirectory under model/models/. Adding one means:

1. Implement Model (forward pass, tensor loading, vocab) in model/models/<name>/model.go.
2. Add a converter under convert/ so safetensors checkpoints can become GGUF.
3. If the prompt format isn't a stock Go template, add a renderer in model/renderers/<name>/.
4. If the output format needs structured parsing (tool calls, thinking, harmony), add a parser in model/parsers/.
5. If the tokenizer is new, add it to tokenizer/.

The list is long: gemma2/3/4/3n, llama/llama4, mistral, qwen2/3/3vl/3next/25vl, deepseek2/ocr, glm4moelite/glmocr, gptoss, lfm2/lfm2-vl, llama-adapter/llama4, mllama, mistral-causal, mixtral, nemotron-h, nomicbert, olmo, phi3, commandr, bert. The complete list is ls model/models plus the converters under convert/.

KV cache

kvcache/causal.go implements the standard causal KV cache; kvcache/encoder.go handles encoder caches for vision models. Composition through kvcache/wrapper.go lets a model layer multiple cache types.

OLLAMA_KV_CACHE_TYPE and OLLAMA_FLASH_ATTENTION (in envconfig/config.go) toggle the runtime behavior; defaults are picked per model.

Sampling

sample/ hosts temperature, top-k, top-p, repeat penalty, and mirostat (kept around even though mirostat, mirostat_tau, mirostat_eta are deprecated as Modelfile parameters — see parser/parser.go).

Tokenizers

tokenizer/ hosts BPE, sentencepiece, and tokenizer-specific code. Recent fixes like tokenizer: fix multi-regex BPE offset handling (#15844) show this module gets careful attention.

Integration points

• Owned by ollamarunner (runner/ollamarunner/); not used by llamarunner (which delegates to llama.cpp).
• The renderer/parser machinery is also used outside the runner — the daemon's prompt assembly path in server/renderer_resolution.go looks up the renderer per model so requests get the right prompt format even when llamarunner does the actual inference.
• Capability checks (types/model/capabilities.go) work against the architecture metadata that this engine consumes.

Entry points for modification

• New architecture → see the five-step list above.
• New backend (e.g., a new accelerator) → implement Backend under ml/backend/.
• New sampler → add it to sample/ and wire it into the option parsing.
• New cache layout → extend kvcache.Cache.

Key source files

File	Purpose
ml/backend.go	Backend, SystemInfo, BackendMemory.
ml/device.go	Device discovery types shared with discover/.
model/model.go	Model interface and helpers.
model/renderers/	Prompt builders.
model/parsers/	Output parsers.
kvcache/cache.go	KV cache interface.
sample/samplers.go	Decoding strategies.
tokenizer/tokenizer.go	Tokenizer entry point.