comfy/

The diffusion engine. By line count (84,388) it dominates the repository. Everything that runs PyTorch lives under here: model architectures, sampling loops, text encoders, VAEs, LoRA, hooks, ControlNet, memory management, and quantization. nodes.py and comfy_extras/*.py import from comfy/ to do their actual ML work.

Layout

comfy/
├── sd.py                       # Top-level loaders: load_checkpoint_guess_config, load_clip, …
├── model_base.py               # BaseModel + per-architecture subclasses
├── supported_models.py         # Registry of model architectures with key/shape detectors
├── model_detection.py          # Probes a state_dict to pick a supported_models entry
├── model_management.py         # VRAM scheduler, dtype/attention probes, device dispatch
├── model_patcher.py            # ModelPatcher: weight ownership + LoRA/hook patches + offload
├── model_sampling.py           # Continuous-time sampling (EDM, FlowMatch, etc.)
├── ops.py                      # Drop-in replacement for torch.nn used by all model code
├── quant_ops.py                # QuantizedTensor + layout registry (FP8 etc.)
├── samplers.py                 # The high-level sampling loop and CFG
├── sample.py                   # Convenience wrappers used by KSampler nodes
├── sampler_helpers.py          # Conds preprocessing for the sampling loop
├── extra_samplers/             # uni_pc, restart, etc.
├── k_diffusion/                # k-diffusion samplers (Euler, DPM++, …) and schedules
├── conds.py                    # Conditioning utilities
├── controlnet.py               # ControlNet, T2I-Adapter loading and apply paths
├── cldm/                       # ControlNet/MMDIT control modules
├── t2i_adapter/                # T2I-Adapter implementation
├── lora.py, lora_convert.py    # LoRA loading + format conversion
├── hooks.py                    # Hook (scoped patch) framework
├── patcher_extension.py        # Extension hooks: callbacks, wrappers, injections
├── float.py                    # Float-cast helpers (FP8/BF16 tricks)
├── pinned_memory.py            # Pinned host memory pool
├── nested_tensor.py            # Nested-tensor helpers
├── rmsnorm.py                  # RMSNorm
├── windows.py                  # Window helpers (Hann, etc.) for audio/video
├── pixel_space_convert.py      # Pixel ↔ latent conversion helpers
├── latent_formats.py           # Per-architecture latent normalization / scale factors
├── memory_management.py        # Lower-level memory primitives (alongside model_management)
├── options.py                  # `args_parsing` flag (mainly for tests)
├── cli_args.py                 # argparse spec for every runtime flag
├── utils.py                    # General helpers (load_torch_file, etc.)
├── diffusers_load.py           # Load diffusers-format checkpoints
├── diffusers_convert.py        # Convert diffusers ↔ comfy weight names
├── clip_model.py               # OpenCLIP backbone used by CLIP/CLIPVision
├── clip_vision.py              # CLIPVision wrapper
├── sd1_clip.py, sdxl_clip.py   # SD1/SDXL CLIP wrappers
├── gligen.py                   # GLIGEN
├── context_windows.py          # Sliding-window context for video
├── ldm/                        # Per-architecture network code (flux, wan, hunyuan_video, …)
├── text_encoders/              # Per-architecture text encoders + tokenizers
├── audio_encoders/             # Audio encoders (whisper, wav2vec2)
├── image_encoders/             # Image encoders (e.g. SigLIP variants)
├── weight_adapter/             # LoRA-like weight adapter layouts
├── taesd/                      # TAESD decoder for previews
├── comfy_types/                # Public type hints (re-exported via the comfy_api)
└── sd1_tokenizer/              # SD1 tokenizer files (BPE, vocab, special tokens)

Key abstractions

Type / function	File	What it is
load_checkpoint_guess_config	comfy/sd.py	The main checkpoint loader. Returns (model, clip, vae, …)
BaseModel + subclasses	comfy/model_base.py	One subclass per architecture. Wraps a diffusion_model (the unet equivalent)
ModelXXX registry (e.g. SDXL, Flux, Wan)	comfy/supported_models.py	Fingerprints for picking the right BaseModel subclass
model_config_from_unet	comfy/model_detection.py	Probes a state dict and matches it against supported_models
ModelPatcher	comfy/model_patcher.py	Owns weights, applies patches lazily, moves model between CPU and GPU
VRAMState, vram_state, cpu_state	comfy/model_management.py	The VRAM scheduling state machine
disable_weight_init, manual_cast_class, MixedPrecisionOps	comfy/ops.py	The torch.nn replacement modes used at load time
QuantizedTensor, QuantizedLayout	comfy/quant_ops.py	Tensor subclass + layout registry for quantization
KSampler, KSAMPLER, sampling_function	comfy/samplers.py	The CFG loop and its high-level wrapper
KSampler (k-diffusion side)	comfy/k_diffusion/sampling.py	Euler, DPM++, Heun, etc.
Hook, HookGroup	comfy/hooks.py	Scoped, time-bounded patches
load_lora, model_lora_keys_unet	comfy/lora.py	LoRA loading and keymap generation
ControlNet, T2IAdapter	comfy/controlnet.py	The two control-style families
model_sampling.ModelSamplingDiscrete etc.	comfy/model_sampling.py	Sampling-time formulations (eps, v, x0, FlowMatch, …)

How a checkpoint becomes a working model

graph TD
    File[(safetensors)] -->|load_torch_file| Tensors[state_dict]
    Tensors -->|model_config_from_unet| Detect[comfy/model_detection.py]
    Detect -->|matches| SM[supported_models.SDXL/Flux/Wan/...]
    SM --> Cfg[model config]
    Cfg --> Build[BaseModel subclass init]
    Build --> Net[diffusion_model<br/>comfy/ldm/&lt;arch&gt;/]
    Net --> Patcher[ModelPatcher]
    Patcher -->|to(device)| MM[model_management]
    Tensors --> CLIPLoad[CLIP loader<br/>comfy/sd1_clip.py + text_encoders/]
    Tensors --> VAELoad[VAE loader<br/>comfy/ldm/&lt;arch&gt;/vae*]

The detection step is the trickiest piece: comfy/model_detection.py reads tensor names and shapes from the state dict and matches them against fingerprints declared on supported_models.ModelXXX classes. New architectures need a new entry there.

Sub-areas worth their own pages

The diffusion engine breaks into several distinct subsystems. Each is documented separately under Systems:

• Sampling pipeline — samplers.py, k_diffusion/, extra_samplers/, model_sampling.py
• Conditioning and CLIP — sd1_clip.py, text_encoders/, conds.py
• Model management — model_management.py, model_patcher.py, memory_management.py, pinned_memory.py, ops.py
• LoRA and hooks — lora.py, lora_convert.py, hooks.py, weight_adapter/, patcher_extension.py
• Quantization — quant_ops.py, MixedPrecisionOps in ops.py, QUANTIZATION.md
• Supported models — what's in supported_models.py, model_base.py, ldm/, text_encoders/

Integration points

• Imported by nodes.py and almost every comfy_extras/*.py file.
• Imported by comfy_api/latest/_io.py for type definitions surfaced to custom nodes.
• Reads comfy.cli_args.args directly — flags propagate through the whole engine via that singleton.
• Calls folder_paths.get_full_path to resolve checkpoint locations.

Where to start a change

• Add a new model architecture: add comfy/ldm/<model>/, register a class in comfy/supported_models.py, add a text encoder under comfy/text_encoders/, add a BaseModel subclass in comfy/model_base.py, and wire nodes in comfy_extras/nodes_<model>.py. The most recent additions (Z Image, Flux 2, LongCat) are the best templates.
• Add a new sampler: add to comfy/k_diffusion/sampling.py or comfy/extra_samplers/. Surface it in comfy/samplers.py's SAMPLER_NAMES.
• Add a new attention backend: extend the dispatch in comfy/ldm/modules/attention.py. The CLI flags (--use-pytorch-cross-attention, --use-flash-attention, --use-sage-attention, …) are wired in comfy/model_management.py.
• Touch memory management: tread carefully. model_management.py and model_patcher.py are top-10 churn files for a reason — they have many subtle device/dtype paths and a sizable test surface in tests-unit/comfy_test/.