Transformers

🤗 Transformers is the model-definition framework for state-of-the-art machine learning across text, vision, audio, video, and multimodal domains. It provides a single PyTorch-based API where 460+ model architectures are implemented faithfully to their original papers and made available for both inference and training.

The library acts as the pivot of the open-source ML ecosystem: when a model definition lands in transformers, downstream training frameworks (Axolotl, Unsloth, DeepSpeed, FSDP, PyTorch-Lightning), inference engines (vLLM, SGLang, TGI), and adjacent runtimes (llama.cpp, mlx) consume that definition. Over 1M+ checkpoints on the Hugging Face Hub load through transformers.

What this wiki covers

Section	What you will find
Architecture	The high-level layering: PreTrainedConfig → PreTrainedModel → Pipeline/Trainer/generate.
Getting started	Install, run a Pipeline, fine-tune, and the major commands (make style, make fix-repo).
Glossary	Project-specific vocabulary (modular, copies, auto classes, kernels, expert parallelism, …).
By the numbers	Quantitative snapshot: 4,300 Python files, 22.7K commits, 462 model directories.
Lore	Eight years of evolution from pytorch-pretrained-bert to v5.
Systems	Configuration, modeling, generation, tokenization, pipelines, trainer, quantization, integrations.
Features	Auto classes, from_pretrained, modular conversion, continuous batching, tensor parallelism, chat templates, serving.
Models	Tour of the 462 model directories, organized by modality.
Background	Design philosophy, V5 migration, deprecations.
Reference	Configuration files, dependencies, repo layout.

The library at a glance

Every supported model exposes the same triplet:

1. A configuration — hyperparameters such as hidden size, number of layers, vocab size (PreTrainedConfig in src/transformers/configuration_utils.py).
2. A model — a torch.nn.Module subclass wrapped by PreTrainedModel (src/transformers/modeling_utils.py, ~5K LOC).
3. A preprocessor — a tokenizer, image processor, video processor, feature extractor, or multimodal processor.

Every triplet supports the same three Hub methods: from_pretrained(), save_pretrained(), and push_to_hub(). The two top-level conveniences sit on top of this triplet:

• Pipeline wraps "preprocess → model → postprocess" for 25+ task types.
• Trainer (src/transformers/trainer.py, 4.4K LOC) provides a full training loop with mixed precision, FSDP, DeepSpeed, gradient checkpointing, and distributed orchestration.

Quick example

from transformers import pipeline

pipe = pipeline(task="text-generation", model="Qwen/Qwen2.5-1.5B")
pipe("the secret to baking a really good cake is ")

Design tenets

The library follows eight principles spelled out in docs/source/en/philosophy.md:

Source of Truth · One Model, One File · Code is the Product · Standardize, Don't Abstract · DRY* (only when it helps users) · Minimal User API · Backwards Compatibility · Consistent Public Surface

The most distinctive of these is One Model, One File: every architecture's core inference and training logic is readable top-to-bottom in a single modeling_<name>.py, even at the cost of duplicated code. The newer modular_<name>.py mechanism keeps that user-facing file untouched while letting maintainers express it as a small diff against another model.

Where to read next

• New to the codebase? Start with Architecture and Getting started.
• Implementing a new model? Read Modular models and Patterns and conventions.
• Debugging generation? Read Generation and KV cache.
• Looking for a specific architecture? Browse Models.