LoRA serving

Active contributors: Jee Jee Li, Antoni Baum, Cyrus Leung.

Purpose

vLLM supports serving many LoRA adapters concurrently against a single base model. Adapters are loaded on demand, kept in a bounded LRU cache, applied per request via the punica kernel family, and updated without restarting the engine. Both dense and MoE LoRA are supported.

Where it lives

vllm/lora/
├── __init__.py
├── lora_model.py         # LoRAModel: holds tensors for one adapter
├── lora_weights.py       # LoRA weight tensor utilities
├── model_manager.py      # LoRAModelManager: cache, scheduling, swapping (~40 KB)
├── worker_manager.py     # LoRAWorkerManager: per-rank loaded adapters
├── request.py            # LoRARequest dataclass
├── resolver.py           # Locates adapter weights (filesystem / HF hub)
├── peft_helper.py        # PEFT-format support
├── utils.py
├── layers/               # LoRA-aware versions of linear, embedding, attention layers
├── ops/                  # Punica wrappers (Triton + CUDA)
└── punica_wrapper/       # Per-shape punica kernel selection

vllm/model_executor/layers/fused_moe/
├── lora_experts_mixin.py # MoE LoRA experts mixin
└── lora_context.py        # Per-call LoRA context for MoE

vllm/plugins/lora_resolvers/
├── filesystem_resolver.py # `lora_filesystem_resolver` plugin
└── hf_hub_resolver.py     # `lora_hf_hub_resolver` plugin

Key abstractions

Abstraction	File	Role
LoRARequest	vllm/lora/request.py	Per-request adapter selection (lora_int_id, lora_path, lora_name)
LoRAModel	vllm/lora/lora_model.py	One adapter's A/B tensors per layer
LoRAModelManager	vllm/lora/model_manager.py	LRU cache + apply/remove/pin
LoRAWorkerManager	vllm/lora/worker_manager.py	Worker-side equivalent (per rank)
BaseLoraResolver	vllm/lora/resolver.py	Pluggable adapter location resolver
PunicaWrapper	vllm/lora/punica_wrapper/	Picks the right Punica kernel for the active shape
LoRA layers	vllm/lora/layers/	LoRA-aware variants of linear, embedding, etc.

How a request gets a LoRA applied

sequenceDiagram
    participant API as API server
    participant Async as AsyncLLM
    participant Mgr as LoRAModelManager (engine)
    participant Res as LoraResolver
    participant Exec as Executor
    participant W as LoRAWorkerManager (workers)
    participant Layer as LoRA-aware layers

    API->>Async: generate(..., lora_request=LoRARequest(name, id, path))
    Async->>Mgr: ensure adapter is loaded
    Mgr->>Res: resolve(name)
    Res-->>Mgr: local path / HF refs
    Mgr->>Mgr: load tensors → LoRAModel
    Mgr->>Exec: add_lora(LoRARequest)
    Exec->>W: collective_rpc("add_lora", ...)
    Async->>Async: tag EngineCoreRequest with lora_int_id
    Note over Layer: per-step forward
    Async->>Exec: schedule_step(...)
    Layer->>Layer: ForwardContext active_lora_ids → punica kernel

Per-step LoRA application uses vllm/lora/punica_wrapper/ to pick the most efficient kernel:

• bgmv — batched grouped matrix-vector for many small adapters.
• sgmv — sequence-major grouped matrix-vector for a few adapters spanning many tokens.
• per-rank decoding kernels for cases that don't fit cleanly into bgmv/sgmv.

The Punica kernels live in vllm/lora/ops/ and are exposed via vllm/_custom_ops.py.

Resolvers

A resolver locates an adapter given its lora_name. Two ship with vLLM as Python entry points (pyproject.toml):

[project.entry-points."vllm.general_plugins"]
lora_filesystem_resolver = "vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver"
lora_hf_hub_resolver = "vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver"

Custom resolvers can be added the same way (subclass BaseLoraResolver, register from a plugin).

MoE LoRA

MoE LoRA replaces the experts' linear projections, not the router. Implementation:

• lora_experts_mixin.py — adds LoRA bookkeeping to FusedMoE.
• lora_context.py — pushes per-call active adapter state into the MoE forward.
• Layer files under vllm/lora/layers/ for embedding, vocab, linear, attention.

Key source files

File	Purpose
vllm/lora/model_manager.py	Engine-side cache, swap, pin
vllm/lora/worker_manager.py	Worker-side state
vllm/lora/request.py	LoRARequest dataclass
vllm/lora/resolver.py	Resolver interface
vllm/lora/peft_helper.py	PEFT format adapter
vllm/lora/punica_wrapper/__init__.py	Per-shape kernel dispatch
vllm/lora/ops/	Triton/CUDA Punica wrappers
vllm/v1/worker/lora_model_runner_mixin.py	Plumbs LoRA into the GPU runner
vllm/model_executor/layers/fused_moe/lora_experts_mixin.py	MoE LoRA experts

Entry points for modification

• New resolver: subclass BaseLoraResolver, register via the vllm.general_plugins entry point (or pass as --lora-resolver).
• Adapter cache policy: tweak the LRU policy in LoRAModelManager.
• Punica kernel: add a kernel under vllm/lora/ops/, surface via _custom_ops.py, route via PunicaWrapper.
• New layer that needs LoRA: add a LoRA-aware variant in vllm/lora/layers/ and register it where the base layer is constructed.

For how layers consume LoRA inside the forward pass, see Model executor. For how scheduler accounts for adapter swap latency, see Scheduler.