LoRA and hooks

How ComfyUI applies LoRA, IP-Adapter, hypernetwork, and other "patch this model temporarily" extensions without copying or reloading the underlying weights.

Purpose

Generative pipelines rely heavily on adapters: LoRAs and their variants (LoCon, LoHa, LoKr, GLora), IP-Adapter, hypernetworks, ControlNet auxiliaries. ComfyUI's design choice is to keep the base model untouched and apply patches lazily through the ModelPatcher, so any number of LoRAs can be combined, weighted, and even time-scoped without duplicating model memory.

Layout

comfy/
├── lora.py                   # Load / convert / apply LoRA
├── lora_convert.py           # Tensor-name remapping (kohya ↔ comfy ↔ diffusers)
├── hooks.py                  # Hook framework (scoped patches)
├── patcher_extension.py      # Callbacks, wrappers, injections (used by hooks/loras)
├── model_patcher.py          # The host that applies all of the above
└── weight_adapter/           # Layout-specific adapter implementations
    ├── __init__.py           # WeightAdapterBase, register/dispatch
    ├── lora.py, locon.py, loha.py, lokr.py, glora.py, oft.py, …

Key abstractions

Type / function	File	What it is
load_lora	comfy/lora.py	Reads a LoRA state dict and produces a key-mapped patch dictionary
convert_lora	comfy/lora_convert.py	Renames tensors so kohya/diffusers/Comfy variants all hit the same target keys
model_lora_keys_unet, model_lora_keys_clip	comfy/lora.py	Per-architecture key maps used to translate a LoRA's keys into ComfyUI parameter paths
add_patches	ModelPatcher (model_patcher.py)	Register a set of patches with a strength
Hook, HookGroup	comfy/hooks.py	A patch with extra scope: timestep range, conditioning ID, etc.
WeightAdapterBase and subclasses	comfy/weight_adapter/	Layout-specific code: how to recompose an adapter into a delta tensor
CallbacksMP, WrappersMP, PatcherInjection	comfy/patcher_extension.py	Extension points the patcher publishes for callbacks, function wrappers, and tree injects

Loading a LoRA

sequenceDiagram
    participant Node as LoraLoader
    participant Conv as comfy.lora_convert
    participant Lora as comfy.lora
    participant MP as ModelPatcher
    Node->>Conv: convert_lora(state_dict)
    Conv-->>Node: state_dict (renamed)
    Node->>Lora: load_lora(state_dict, key_map_unet + key_map_clip)
    Lora-->>Node: { target_key: (rank, up, down, alpha, …) }
    Node->>MP: clone() then add_patches(patches, strength_model)
    Node->>MP: also add_patches on the CLIP patcher with strength_clip

load_lora_for_models (the actual node function in comfy/sd.py) is the orchestrator. It clones both the model and CLIP patchers — clones are cheap because they share weights — and then registers the LoRA as patches on the clones. The next sample on those clones applies the patches lazily as weights stream to GPU.

Weight adapter dispatch

A LoRA file might be one of many layouts: classic LoRA (rank-decomposed), LoCon (extends to convolutions), LoHa (Hadamard product), LoKr (Kronecker), GLora, OFT, and more. The comfy/weight_adapter/ directory has a class per layout:

• WeightAdapterBase — the interface (compose to delta, apply to base weight, etc.)
• Subclasses: LoRAAdapter, LoConAdapter, LoHaAdapter, LoKrAdapter, GLoraAdapter, OFTAdapter.

load_lora inspects each tensor block in the state dict and picks the matching adapter. The result is a unified list of patches that the ModelPatcher applies the same way regardless of layout.

Hooks

Hooks are LoRA's bigger sibling. A Hook is also a patch on a ModelPatcher, but it carries scope:

• Timestep range — apply only between sigma_a and sigma_b.
• Conditioning binding — apply only for specific conditioning entries (so positive and negative prompts can have different LoRAs).
• Hook type — weight patches, attention patches, sampling-time callbacks.

HookGroup aggregates hooks; the sampler calls apply_hooks_for_timestep (in comfy/hooks.py) before each step to install/uninstall the right set. The user surface is in comfy_extras/nodes_hooks.py.

Patcher extensions

patcher_extension.py exposes three extension points used by hooks, LoRAs, and several comfy_extras nodes:

• Callbacks (CallbacksMP) — fire before/after specific phases (apply_model, sampling).
• Wrappers (WrappersMP) — wrap a function call to inject pre/post logic. Used heavily for PAG/SAG/CFG variants in comfy_extras/.
• Injections (PatcherInjection) — splice extra nn.Modules into the model graph at a known site. Used for adapters that need their own modules rather than weight deltas.

These are the reason you can stack many comfy_extras "guidance" nodes (PAG + SAG + APG + …) without them stepping on each other — each registers itself as a wrapper.

Hypernetwork and IP-Adapter

• Hypernetwork — comfy_extras/nodes_hypernetwork.py loads classic hypernetwork weights and applies them via the same patch mechanism on attention layers.
• IP-Adapter — implemented in comfy_extras/nodes_model_patch.py (the file is 30 KB) using PatcherInjection to add cross-attention modules driven by CLIP_VISION_OUTPUTs.

Integration points

• LoRA: invoked by LoraLoader and LoraLoaderModelOnly nodes in nodes.py.
• Hooks: invoked by comfy_extras/nodes_hooks.py and consumed by Sampling pipeline (samplers.py calls apply_hooks_for_timestep).
• Weight adapters: invoked transitively from load_lora.
• Patcher extensions: used by many comfy_extras modules — the pattern is "node clones the model, registers a wrapper/callback, returns the clone."

Where to start a change

• Adding a new LoRA layout: subclass WeightAdapterBase in comfy/weight_adapter/, register it. The dispatcher in load_lora will pick it up if the layout-detection logic in weight_adapter/__init__.py accepts it.
• Adding a new hook type: extend Hook in comfy/hooks.py; make sure the sampler calls into the new code path.
• A guidance variant that wants to wrap a sampler step: register a WrappersMP entry rather than editing samplers.py. Look at comfy_extras/nodes_pag.py as a template.