Conditioning and CLIP

How text prompts become tensors that condition the diffusion model. Despite the name, the CLIP slot in ComfyUI carries any text encoder — CLIP itself, T5, Llama, Qwen 2.5, and more — depending on the model architecture.

Purpose

Turn a string prompt into a CONDITIONING value: a list of (tensor, dict) tuples where the tensor is a per-token embedding and the dict carries optional pooled outputs, areas, masks, strength, timestep ranges, ControlNet refs, and hooks.

Layout

comfy/
├── sd1_clip.py                # Generic CLIP wrapper used as the parent for many encoders
├── sdxl_clip.py               # SDXL dual-CLIP wrapper
├── clip_model.py              # OpenCLIP backbone
├── clip_vision.py             # CLIPVision wrapper (image → embedding)
├── conds.py                   # Conditioning data utilities
└── text_encoders/
    ├── flux.py                # CLIP-L + T5 XXL
    ├── sd3_clip.py            # CLIP-G + CLIP-L + T5 XXL
    ├── hunyuan_video.py       # CLIP-L + Llama 3
    ├── qwen_image.py, qwen_vl.py, qwen35.py
    ├── llama.py               # Llama 3 + tokenizer
    ├── lumina2.py             # Gemma
    ├── hidream.py             # CLIP-L + CLIP-G + T5 XXL + Llama
    ├── ace.py, ace15.py       # ACE Step audio encoders
    ├── kandinsky5.py, jina_clip_2.py, sam3_clip.py
    ├── t5.py                  # Generic T5 wrapper
    ├── … plus tokenizer JSONs and BPE/Sentencepiece configs

There are 30+ files in comfy/text_encoders/ — one per family supported by the engine.

Key abstractions

Type / function	File	What it is
CLIP (comfy.sd.CLIP)	comfy/sd.py	The high-level handle — wraps a cond_stage_model (the actual encoder) + a tokenizer
cond_stage_model	comfy/sd1_clip.py and friends	The PyTorch encoder. Wrapped in a ModelPatcher for offloading
tokenize	CLIP.tokenize	String → list of (token_id, weight) per stream
encode_from_tokens_scheduled	CLIP.encode_from_tokens_scheduled	Returns the conditioning tensor list
CLIPTextEncode	nodes.py	The user-facing node
ConditioningCombine, ConditioningSetArea, ConditioningSetMask, ConditioningSetTimestepRange	nodes.py	Manipulate the conditioning list
CLIPVisionLoader, CLIPVisionEncode	nodes.py	Image → embedding for SD-Style and IP-Adapter-style use

How a prompt becomes conditioning

graph LR
    Prompt["text<br/>(positive prompt:1.2) (negative:0.8)"] --> Tok[CLIP.tokenize]
    Tok --> Tokens["[(t1, w1), (t2, w2), ...]"]
    Tokens --> Enc[CLIP.encode_from_tokens_scheduled]
    Enc --> Cond["[(tensor, {pooled_output, ...}), ...]"]
    Cond --> Node[CLIPTextEncode output]
    Node --> Down[Downstream nodes:<br/>ConditioningCombine,<br/>ConditioningSetArea,<br/>ControlNet apply,<br/>...]
    Down --> KSampler[KSampler]

The (token, weight) step lets ComfyUI honor (word:1.2) weight syntax. The frontend also handles dynamic prompts ({a|b|c}) before sending. Comments (// ... and /* ... */) are stripped client-side.

Multi-encoder models

Most modern models use more than one text encoder concatenated:

Architecture	Encoders
SDXL	CLIP-L + CLIP-G
SD3 / SD3.5	CLIP-L + CLIP-G + T5 XXL
Flux	CLIP-L + T5 XXL
Hunyuan Video	CLIP-L + Llama 3
HiDream	CLIP-L + CLIP-G + T5 XXL + Llama
Lumina 2	Gemma
Qwen Image	Qwen 2.5 VL
ACE Step	Custom audio-aware encoder
Wan 2.x	UMT5

Loading happens via CLIPLoader (single encoder) or DualCLIPLoader/TripleCLIPLoader/QuadrupleCLIPLoader in nodes.py. The loaders dispatch into comfy.sd.load_clip which builds the right combination based on the model type.

Conditioning ops

The conditioning list is just a list of (tensor, dict) pairs, so manipulating it is straightforward:

• ConditioningCombine — concatenate two lists (treats each as a "stream").
• ConditioningAverage — weighted blend between two lists.
• ConditioningSetArea — set area and strength on each item.
• ConditioningSetMask — set mask on each item.
• ConditioningSetTimestepRange — set timestep_start/timestep_end.
• ConditioningConcat — concatenate along the token dimension (rather than the list).

These are all in nodes.py and they all just edit the dict alongside each tensor.

CLIP Vision

Image encoders for SD-style image conditioning. CLIPVisionLoader loads a SigLIP/CLIP vision model (comfy/clip_vision.py); CLIPVisionEncode returns a CLIP_VISION_OUTPUT carrying the pooled output and last hidden state.

The style_models/ folder type holds (T2I-Adapter style and CSGO-style) style models that consume CLIP_VISION_OUTPUTs.

Tokenization details

Each encoder has a tokenizer:

• BPE (CLIP) — the SD1 tokenizer files live in comfy/sd1_tokenizer/.
• Sentencepiece (T5, UMT5, Gemma) — comfy/text_encoders/spiece_tokenizer.py.
• Llama — comfy/text_encoders/llama_tokenizer/.
• Qwen 2.5 — comfy/text_encoders/qwen25_tokenizer/.
• Qwen 3.5 — comfy/text_encoders/qwen35_tokenizer/.
• HyDiT BERT — comfy/text_encoders/hydit_clip_tokenizer/.
• byT5 (small glyph) — comfy/text_encoders/byt5_tokenizer/.
• ACE lyrics — comfy/text_encoders/ace_lyrics_tokenizer/ plus a custom cleaner in ace_text_cleaners.py.
• t5_pile — comfy/text_encoders/t5_pile_tokenizer/.

Token weights work via the schedule format inside encode_from_tokens_scheduled: tokens with weight ≠ 1.0 are encoded into a separate stream and blended back in.

Embeddings (textual inversion)

Strings of the form embedding:my_concept.pt in the prompt are resolved to files under models/embeddings/ and substituted into the token stream by CLIP.tokenize. Implementation lives in comfy/sd1_clip.py.

Integration points

• Loaded by comfy.sd.load_checkpoint_guess_config (which builds a CLIP from the same checkpoint that supplies the unet) or by standalone CLIPLoader nodes.
• Consumed by Sampling pipeline — every cond in the conditioning list ends up in apply_model's c_crossattn (or per-model equivalent).
• LoRA can patch the text encoder; see LoRA and hooks.

Where to start a change

• Adding a new encoder family: add a file under comfy/text_encoders/, add a tokenizer, register it in comfy.sd.load_clip. Most existing files are 30-200 lines and follow the same template.
• Adding a new prompt syntax: extend comfy/sd1_clip.py's tokenize path. The frontend may also need updates for dynamic prompts.
• New conditioning op: add a node to nodes.py or comfy_extras/nodes_cond.py. Just produce a new list and never mutate the input list.