Processing

Purpose

Processing covers everything that turns raw inputs (text, image, audio, video, or a mix of those) into model-ready tensors. Different modalities have different preprocessing needs, so the library uses several base classes that all share the same from_pretrained / save_pretrained / push_to_hub contract as configurations and tokenizers.

Key abstractions

Class	File	Modality
ProcessorMixin	src/transformers/processing_utils.py (101K LOC)	Multimodal: bundles tokenizer + image/feature/video processor
BaseImageProcessor, BaseImageProcessorFast	src/transformers/image_processing_base.py (23K LOC), src/transformers/image_processing_utils.py (29K LOC), src/transformers/image_processing_backends.py (27K LOC)	Image
BaseVideoProcessor	src/transformers/video_processing_utils.py (39K LOC)	Video
SequenceFeatureExtractor, FeatureExtractionMixin	src/transformers/feature_extraction_utils.py (30K LOC), src/transformers/feature_extraction_sequence_utils.py (19K LOC)	Audio
image_transforms	src/transformers/image_transforms.py (45K LOC)	Resize, normalize, pad, center-crop helpers
image_utils, video_utils, audio_utils	top-level files	Loaders and IO helpers

The auto-class equivalents are AutoImageProcessor, AutoVideoProcessor, AutoFeatureExtractor, AutoProcessor (in src/transformers/models/auto/).

Image processors

Two implementation tiers:

• Slow (BaseImageProcessor): pure NumPy / PIL.
• Fast (BaseImageProcessorFast): torchvision.transforms.v2-backed, batch-friendly, GPU-capable.

The fast processors became the default for many models in 2024-2025. They support tensor inputs directly and can run on CUDA when given CUDA tensors.

The library has 194 image_processing_*.py files. Each typically declares the model-specific resize, normalize, and split-into-patches logic.

Feature extractors

Used for audio (raw waveform → log-mel spectrogram, MFCCs) and certain time-series models. Per-model files: feature_extraction_<arch>.py.

Video processors

Introduced in 2024. Decode video to a fixed number of frames and apply the model's frame-level transforms. Per-model files: video_processing_<arch>.py.

Multimodal Processor

For models like CLIP, BLIP, LLaVA, Idefics, Qwen-VL, etc., a <Arch>Processor class subclasses ProcessorMixin and bundles a tokenizer and one or more modality processors. Calling the processor:

processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
inputs = processor(images=image, text="a photo of", return_tensors="pt")
model.generate(**inputs)

ProcessorMixin handles from_pretrained round-trips for all bundled subclasses.

Chat templates with images / audio / video

When a multimodal Processor has a chat template, processor.apply_chat_template(messages, tokenize=False) interpolates image/video/audio placeholders. The matching extension to chat_template_utils.py is in src/transformers/utils/chat_parsing_utils.py.

Where transforms live

• src/transformers/image_transforms.py — resize, normalize, pad, center-crop, color conversions, Tensor ↔ PIL adapters.
• src/transformers/image_utils.py — load images from URLs/paths, validate inputs, compute target sizes.
• src/transformers/video_utils.py — sampling strategies, decoding, codec handling.
• src/transformers/audio_utils.py (55K LOC) — STFT, mel filterbanks, fbank features.

Backbones for vision

Vision models often re-use a backbone (timm, torchvision, dinov2, etc.). The backbone abstraction (BackboneMixin) lives in src/transformers/backbone_utils.py and exposes:

• out_features — list of named output feature maps.
• out_indices — list of integer indices.
• forward_with_filtered_kwargs — filters kwargs to match the backbone's signature.

This is what detection (DETR, RT-DETR), segmentation (Mask2Former), and depth-estimation (DPT, ZoeDepth) models rely on.

Testing

• tests/test_image_processing_common.py (35K LOC) — shared mixin.
• tests/test_processing_common.py (95K LOC) — shared multimodal processor mixin.
• tests/test_video_processing_common.py (25K LOC) — shared video mixin.
• tests/test_feature_extraction_common.py, tests/test_sequence_feature_extraction_common.py — audio mixins.
• tests/test_image_transforms.py (25K LOC) — transform unit tests.

Integration points

• Pipelines in src/transformers/pipelines/ for vision/audio/multimodal tasks (image-classification, image-segmentation, image-text-to-text, automatic-speech-recognition, text-to-audio, video-classification, mask-generation, keypoint-matching, depth-estimation, …) instantiate the right processor via AutoProcessor / AutoImageProcessor / AutoFeatureExtractor / AutoVideoProcessor.
• The Hub I/O layer (src/transformers/utils/hub.py) handles preprocessor_config.json and friends.
• Trainer does not call processors directly; user code prepares a Dataset whose items already pass through the processor.

Entry points for modification

• Add a processor for a new model → drop a <modality>_processing_<name>.py (and/or processing_<name>.py) into src/transformers/models/<arch>/, register in the relevant auto mapping. Add a fast variant when possible.
• Improve transform performance → contribute to src/transformers/image_transforms.py or to the fast-processor base classes.
• Multimodal chat parsing → src/transformers/utils/chat_parsing_utils.py.