Scheduler

A goroutine that decides when to load a model into a runner subprocess, when to keep it loaded, and when to evict it. Lives entirely in server/sched.go.

Purpose

Multiple HTTP requests can arrive concurrently for the same model, for different models, or for cloud-only models. The scheduler keeps the load decisions sequential, the eviction decisions VRAM-aware, and the request handling parallel where it's safe.

Key abstractions

Symbol	Location	Purpose
Scheduler	server/sched.go	The struct. Holds the loaded runners, the channels handlers send requests on, and override hooks for tests.
LlmRequest	server/sched.go	One request to schedule: model, options, keep-alive, success/error channels.
runnerRef	server/sched.go	Reference to a loaded runner: the LlamaServer, refcount, expiration, model.
Scheduler.Run	server/sched.go	The main loop. Reads pendingReqCh, finishedReqCh, expiredCh, unloadedCh.
Scheduler.processPending	server/sched.go	Decides reuse / load / evict.
Scheduler.processCompleted	server/sched.go	Bookkeeping when a request finishes.
Scheduler.GetRunner	server/sched.go	Public entry point: wraps a request in LlmRequest and returns success/error channels.
Scheduler.maybeFindCPURunner / findRunnerToUnload / evictRunner	server/sched.go	Eviction policy.
loadFn, newServerFn, getGpuFn, getSystemInfoFn	server/sched.go	Hooks injected by tests so the scheduler can run without GPUs or subprocesses.

How it works

graph TD
    Handler[Server handler] -->|GetRunner| PendingCh[pendingReqCh]
    PendingCh --> Loop[Scheduler.Run]
    Loop --> Process[processPending]
    Process -->|already loaded| Reuse[refcount++ -> success]
    Process -->|fits in VRAM| Load[loadFn -> spawn runner]
    Process -->|not enough VRAM| Evict[findRunnerToUnload]
    Evict --> Unload[evictRunner -> Close]
    Unload --> Load
    Loop --> Completed[processCompleted]
    Completed -->|refcount==0| Expire[set expiry]
    Loop --> Expired[handle expiredCh]
    Expired --> Unload

Only one model can be loading at a time (activeLoading field) — multiple concurrent loads would compete for the same VRAM and confuse capacity planning. Already-loaded models can serve requests in parallel up to OLLAMA_NUM_PARALLEL per runner.

Decision steps

processPending:

1. Look up the request's model name in loaded. If present, increment refcount and return.
2. If a load is already in progress and it's not for this model, requeue.
3. Otherwise call getGpuFn/getSystemInfoFn to inspect available VRAM and loadFn to start the runner.
4. If loadFn says it cannot fit, look at loaded for a candidate to evict (findRunnerToUnload), close it, and retry.
5. If still no fit (or no GPU), fall back to CPU via maybeFindCPURunner.

loadFn is the seam where the scheduler hands off to llm.NewLlamaServer (or its equivalent for ollamarunner/mlxrunner/imagegen). It returns whether the model fit on the requested devices — if not, the scheduler tries again after evicting.

Keep-alive and eviction

Each runnerRef has an expireAt time. processCompleted resets it when the last in-flight request finishes; an internal timer fires on expiredCh and evictRunner runs. The default keep-alive is 5 minutes (OLLAMA_KEEP_ALIVE); 0 evicts immediately, negative values keep models loaded indefinitely.

VRAM pressure also triggers eviction. findRunnerToUnload prefers idle runners over busy ones and the oldest idle runner over newer ones.

Bounded queues

The scheduler's channels are sized to envconfig.MaxQueue(). When the queue is full, GetRunner returns ErrMaxQueue ("server busy, please try again. maximum pending requests exceeded") and the handler responds 503.

Defaults

• defaultModelsPerGPU = 3 — cap on how many models the scheduler will try to coexist on a single GPU before evicting.
• waitForRecovery = 5 * time.Second — grace period to let a misbehaving runner exit cleanly before the scheduler considers it lost.

Integration points

• Spawned by server.NewServer and stored on Server.sched.
• Calls into discover.GPUDevices and discover.GetSystemInfo for hardware data.
• Uses fs/ggml/ and fs/gguf/ to inspect model metadata when computing fit.
• Hands off to llm.NewLlamaServer for the actual subprocess spawn.

Entry points for modification

• New eviction policy → adjust findRunnerToUnload. Add a test in server/sched_test.go using the existing loadFn/newServerFn injection pattern.
• New "when to refuse" rule → add it to GetRunner or processPending before Load. Surface it as a typed error.
• Load-balancing across heterogeneous GPUs is currently coarse (model-per-GPU); the place to refine it is processPending plus the discover package.

Key source files

File	Purpose
server/sched.go	Whole scheduler implementation.
server/sched_test.go	Shows the injection points used in tests.
discover/gpu.go	GPUDevices source.
llm/server.go	NewLlamaServer — the default loadFn.