huggingface/transformers

Multimodal models

Models that combine more than one modality on the input or output side. The biggest groups are vision-language models (VLMs), document AI models, and recent any-to-any models.

Vision-language models (image-text-to-text)

Family	Directories	Notes
LLaVA family	`llava/`, `llava_next/`, `llava_next_video/`, `llava_onevision/`, `vipllava/`, `video_llava/`	Encoder + projector + LM
BLIP family	`blip/`, `blip_2/`, `instructblip/`, `instructblipvideo/`	Captioning, VQA
Idefics	`idefics/`, `idefics2/`, `idefics3/`, `smolvlm/`	Open VLMs
Qwen-VL family	`qwen2_vl/`, `qwen2_5_vl/`, `qwen3_vl/`, `qwen3_vl_moe/`	Multi-resolution image + video
GLM-VL	`glm4v/`, `glm4v_moe/`, `glm46v/`, `glm_image/`	THUDM VLMs
InternVL	`internvl/`, `mlcd/`	Open VLMs
Cohere	`cohere2_vision/`
Aya	`aya_vision/`	Multilingual VLM
Pi0 / Perception	`pi0/`, `perception_lm/`, `pe_video/`, `pe_audio/`, `pe_audio_video/`	Multimodal foundation
Florence / Kosmos	`florence2/`, `kosmos2/`, `kosmos2_5/`, `chameleon/`, `emu3/`, `janus/`, `paligemma/`, `mistral3/`	Tile or token-level multimodal
Mllama	`mllama/`	Llama 3.2 vision
Other VLMs	`evolla/`, `fast_vlm/`, `fuyu/`, `aria/`, `colpali/`, `colmodernvbert/`, `colqwen2/`, `mistral4/`, `metaclip_2/`, `pixtral/`, `pixio/`, `bridgetower/`, `vilt/`, `lxmert/`, `flava/`, `chinese_clip/`, `altclip/`, `align/`, `clip/`, `clipseg/`, `siglip/`, `siglip2/`, `groupvit/`, `git/`, `granite_vision`, `glmasr/`, `glm_image/`, `mistral3/`, `glm_ocr/`	Various
Edge / mobile	`edgetam/`, `edgetam_video/`, `smolvlm/`, `mobilevitv2/`
Smaller models	`chmv2/`, `cohere2/`, `dinov2/`, `dinov2_with_registers/`, `dinov3_convnext/`, `dinov3_vit/`, `flamingo-style models`, `mistral4/`, `granite-speech/`

Document understanding

These often combine image + text and produce text or structured outputs.

Directory	Notes
`donut/`, `nougat/`, `udop/`, `pix2struct/`, `markuplm/`, `layoutlm/`, `layoutlmv2/`, `layoutlmv3/`, `layoutxlm/`, `mgp_str/`, `lighton_ocr/`, `pp_chart2table/`, `pp_formulanet/`, `pp_ocrv5_*/`, `qianfan_ocr/`, `glm_ocr/`, `got_ocr2/`, `paddleocr_vl/`, `slanet/`, `slanext/`	Document AI

Any-to-any and unified

Directory	Notes
`chameleon/`, `emu3/`, `janus/`, `qwen2_5_omni/`, `qwen3_omni_moe/`, `phi4_multimodal/`, `audioflamingo3/`, `pi0/`, `perception_lm/`, `pe_audio_video/`, `dia/`, `csm/`, `moshi/`, `glm_moe_dsa/`, `youtu/`, `mistral4/`, `voxtral/`, `voxtral_realtime/`, `seamless_m4t_v2/`	Models that emit text + audio + image / accept arbitrary mixtures

VLMs typically pair a vision tower (often siglip/, siglip2/, clip/, dinov2/, aimv2/, metaclip_2/, mlcd/, or internvl/-bundled) with an LLM. The library exports both halves separately so users can swap encoders.

Multimodal preprocessing

Multimodal models declare a Processor class (processing_<arch>.py) that bundles a tokenizer with one or more modality processors. The base is ProcessorMixin in src/transformers/processing_utils.py (101K LOC). For chat workloads, multimodal chat templates accept content as a list of typed parts (text, image, audio, video). See Chat templates and Processing.

Pipelines that consume multimodal models

Pipeline task	Module
`image-text-to-text`	`src/transformers/pipelines/image_text_to_text.py` (23K LOC)
`document-question-answering`	`src/transformers/pipelines/document_question_answering.py` (29K LOC)
`visual-question-answering` (legacy)	within `image_text_to_text.py`
`any-to-any`	`src/transformers/pipelines/any_to_any.py` (26K LOC)

Continuous batching for VLMs

Many VLMs ship paged-attention support, so transformers serve --continuous-batching works for them too. Constraints: the processor must be deterministic on identical inputs (it is, by design), and the model's _supports_paged_attention flag must be True. See Continuous batching.