LLM subprocess

The llm package is the daemon side of the runner contract. It defines LlamaServer, the interface every runner backend implements, and llmServer, the default implementation that spawns and supervises a runner subprocess.

Purpose

Hide the fact that inference happens in a separate process behind a Go interface. Everything from the scheduler to the chat handlers calls Completion, Embedding, Tokenize, MemorySize, Close — they never see the subprocess directly.

Key abstractions

Symbol	Location	Purpose
LlamaServer	llm/server.go	The interface.
llmServer	llm/server.go	The default implementation.
LoadRequest	llm/server.go	Parameters used to start the runner: model path, projectors, adapters, options.
CompletionRequest / CompletionResponse	llm/server.go	The streaming request/response shape between the daemon and the runner.
StatusWriter	llm/status.go	Captures runner stderr; Close() returns the tail so error responses are useful.
NewLlamaServer	llm/server.go	Default loadFn for the scheduler. Picks a free port, builds the command line, starts the subprocess, waits for /health.

How it works

sequenceDiagram
    participant Sched as Scheduler
    participant LLM as llm.NewLlamaServer
    participant Cmd as exec.Cmd
    participant Runner as ollama runner

    Sched->>LLM: NewLlamaServer(systemInfo, gpus, model, ggml, ...)
    LLM->>LLM: pick free port
    LLM->>LLM: build args (--port, --ctx-size, --threads, --gpu-layers, ...)
    LLM->>Cmd: exec ollama runner ...
    Cmd->>Runner: starts
    LLM->>Runner: GET /health (poll)
    Runner-->>LLM: 200
    LLM-->>Sched: llmServer{ready}
    Sched->>LLM: Load(ctx, ...)
    LLM->>Runner: POST /load
    Runner-->>LLM: ok
    Sched->>LLM: Completion(req, fn)
    LLM->>Runner: POST /completion
    Runner-->>LLM: NDJSON token stream
    LLM-->>Sched: forward via fn
    Sched->>LLM: Close()
    LLM->>Runner: SIGTERM

What the daemon sees

Every interaction with a loaded model goes through LlamaServer:

type LlamaServer interface {
    ModelPath() string
    Load(ctx context.Context, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) ([]ml.DeviceID, error)
    Ping(ctx context.Context) error
    WaitUntilRunning(ctx context.Context) error
    Completion(ctx context.Context, req CompletionRequest, fn func(CompletionResponse)) error
    Embedding(ctx context.Context, input string) ([]float32, int, error)
    Tokenize(ctx context.Context, content string) ([]int, error)
    Detokenize(ctx context.Context, tokens []int) (string, error)
    Close() error
    MemorySize() (total, vram uint64)
    VRAMByGPU(id ml.DeviceID) uint64
    Pid() int
    GetPort() int
    GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
    HasExited() bool
    ContextLength() int
}

The handlers in server/routes.go only ever call this interface; the runner backend is hidden.

Environment passthrough

The filteredEnv helper at the top of llm/server.go decides which environment variables follow the runner subprocess and which don't. It logs OLLAMA_*, CUDA_*, ROC*_*, HIP_*, GPU_*, HSA_*, GGML_*, plus PATH, LD_LIBRARY_PATH, and DYLD_LIBRARY_PATH. Anything else is dropped to keep secrets out of logs.

Stderr capture

Runner stderr is piped through StatusWriter (llm/status.go). It keeps a small ring buffer of the latest output. When the runner exits unexpectedly, Close() returns the tail; the daemon includes that text in the error returned to the API caller, so users see something like "runner process has terminated: cudaMalloc out of memory" instead of an opaque socket error.

Memory accounting

llmServer.MemorySize() and VRAMByGPU(id) report what the runner says it allocated. The scheduler uses these numbers to decide whether more models can fit on the same GPU.

Integration points

• The scheduler (server/sched.go) holds an llmServer per loaded model.
• ml/ supplies the SystemInfo and DeviceInfo structs the daemon passes in.
• fs/ggml/ is used to peek at model metadata to pick context size and parallelism defaults.

Entry points for modification

• A new runner backend with subprocess semantics → write a new LlamaServer implementation, mirror llmServer but spawn your binary, expose the same HTTP API.
• New runtime knob → add it to LoadRequest, surface it through NewLlamaServer's argument parsing, and document it in envconfig/config.go if it's user-facing.

Key source files

File	Purpose
llm/server.go	LlamaServer interface + llmServer implementation.
llm/status.go	Stderr ring buffer for runner crash diagnostics.
llm/llm_darwin.go, llm/llm_linux.go, llm/llm_windows.go	Platform-specific shims (mostly empty on Unix).