Discover

GPU and CPU enumeration. Lives in discover/ and is the only place the rest of the daemon learns about hardware.

Purpose

Tell the scheduler which GPUs exist, how much VRAM each has, what library family they belong to (CUDA, ROCm, Metal), and whether the host has the right drivers loaded. Provide a single source of truth so the scheduler doesn't have to know whether nvml, rocm-smi, or a Metal API call answered the question.

Directory layout

discover/
├── gpu.go                  # GPUDevices entry point, list of detected GPUs
├── gpu_darwin.go           # Metal probing
├── gpu_info_darwin.h, .m   # Objective-C glue for Metal info
├── cpu_linux.go            # /proc/cpuinfo and friends
├── cpu_windows.go          # GetSystemInfo
├── runner.go               # FilteredRunnerDiscovery and helpers
├── types.go                # DeviceInfo, SystemInfo
└── *_test.go

Key abstractions

Symbol	Location	Purpose
GPUDevices	discover/gpu.go	Returns []ml.DeviceInfo for the host.
GetSystemInfo	discover/gpu.go, discover/cpu_*.go	Returns the ml.SystemInfo summary.
FilteredRunnerDiscovery	discover/runner.go	Filters discovered devices to those usable by a specific runner backend.
DeviceInfo, DeviceID (re-exported via ml/device.go)	discover/types.go	The shared shape used by the scheduler and the runner.

How it works

1. discover.GPUDevices(ctx, runners) fans out to platform-specific probes:
   • macOS: Metal info collected through Objective-C in discover/gpu_info_darwin.m.
   • Linux: NVIDIA via NVML, AMD via the ROCm libraries (the runtime libraries are loaded dynamically; missing drivers just mean the GPU isn't reported).
   • Windows: Equivalent NVIDIA / AMD probes plus integrated GPUs where applicable.
2. Each probe returns DeviceInfo records: {ID, Name, Vendor, Family, TotalMemory, FreeMemory, Capabilities}.
3. FilteredRunnerDiscovery lets a runner backend say "I only support these vendors / capabilities," so the scheduler doesn't try to load an MLX model onto a CUDA card.

Integration points

• The scheduler calls getGpuFn (default discover.GPUDevices) on every load decision in server/sched.go.
• getSystemInfoFn (default discover.GetSystemInfo) feeds the SystemInfo struct that the runner subprocess receives via LlamaServer.Load.
• OLLAMA_GPU_OVERHEAD (from envconfig/config.go) reserves VRAM that discover reports as taken so the scheduler avoids edge-of-the-cliff allocations.

Entry points for modification

• Add a new accelerator → write a probe under discover/, populate DeviceInfo, and add it to GPUDevices. Then teach the relevant runner backend to use it.
• Improve VRAM accuracy → most adjustments happen in the platform-specific probes; the scheduler trusts what comes back.

Key source files

File	Purpose
discover/gpu.go	Cross-platform GPU enumeration.
discover/gpu_darwin.go, discover/gpu_info_darwin.m	macOS Metal info.
discover/cpu_linux.go	Linux CPU info from /proc/cpuinfo.
discover/cpu_windows.go	Windows CPU info.
discover/runner.go	Filters devices per runner backend.
discover/types.go	DeviceInfo, SystemInfo.