Architecture

Transformers is layered. At the bottom sit three primitive classes that every architecture implements; in the middle sit cross-cutting subsystems (generation, caching, attention, integrations); at the top sit two high-level user APIs (Pipeline and Trainer). The CLI (transformers chat, transformers serve) is a thin wrapper over these.

Layering

graph TD
    subgraph User["User-facing APIs"]
        Pipeline["Pipeline (src/transformers/pipelines/)"]
        Trainer["Trainer (src/transformers/trainer.py)"]
        Generate["model.generate (src/transformers/generation/)"]
        CLI["transformers CLI (src/transformers/cli/)"]
    end

    subgraph Triplet["Per-model triplet"]
        Config["PreTrainedConfig"]
        Model["PreTrainedModel (torch.nn.Module)"]
        Preproc["Tokenizer / ImageProcessor / FeatureExtractor / Processor"]
    end

    subgraph Cross["Cross-cutting subsystems"]
        Cache["Cache (cache_utils.py)"]
        Attn["Attention dispatcher (modeling_*attention*)"]
        Quant["Quantizers (quantizers/, integrations/)"]
        Hub["Hub I/O (utils/hub.py, core_model_loading.py)"]
        TP["Tensor parallel (integrations/tensor_parallel.py)"]
    end

    Pipeline --> Triplet
    Trainer --> Triplet
    Generate --> Model
    Generate --> Cache
    CLI --> Pipeline
    CLI --> Generate

    Model --> Cache
    Model --> Attn
    Model --> Quant
    Triplet --> Hub
    Model --> TP

The three core primitives

Every model directory under src/transformers/models/<name>/ contains at minimum:

• configuration_<name>.py — a subclass of PretrainedConfig (src/transformers/configuration_utils.py) describing hyperparameters. Configurations are JSON-serialized to config.json on the Hub.
• modeling_<name>.py — one or more subclasses of PreTrainedModel (src/transformers/modeling_utils.py). The base class hosts from_pretrained, save_pretrained, push_to_hub, weight tying, gradient checkpointing, attention dispatch, FSDP/TP integration, and quantization hooks (the file is ~5,000 LOC).
• tokenization_<name>.py and/or image_processing_<name>.py / feature_extraction_<name>.py / video_processing_<name>.py / processing_<name>.py for multimodal models.

There are 462 model directories, 445 modeling_*.py files, 230 modular_*.py shards, 99 tokenizer files, and 194 image-processor files in src/transformers/models/.

Auto classes — late binding

Most users do not instantiate LlamaForCausalLM directly. Instead they call AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B"). The auto package (src/transformers/models/auto/) maintains lookup tables (auto_mappings.py, configuration_auto.py, modeling_auto.py, etc.) that map a config's model_type field to the right concrete class. See Auto classes.

How a forward pass becomes generation

sequenceDiagram
    participant U as User
    participant T as Tokenizer
    participant M as PreTrainedModel
    participant G as generate (generation/utils.py)
    participant C as Cache
    participant L as LogitsProcessor

    U->>T: text
    T->>M: input_ids, attention_mask
    U->>G: generate(**inputs, max_new_tokens=128)
    loop for each new token
        G->>M: forward(input_ids, past_key_values, attention_mask)
        M->>C: read/write KV state
        M-->>G: logits
        G->>L: process_logits (temperature, top-k, repetition penalty)
        L-->>G: filtered logits
        G->>G: sample / argmax
    end
    G-->>U: generated_ids
    U->>T: decode(generated_ids)
    T-->>U: text

generate (src/transformers/generation/utils.py, 3,887 LOC) is the most-used decoding entry point. It supports greedy, beam, contrastive, sample, assisted (speculative), and DoLa decoding, plus continuous batching via src/transformers/generation/continuous_batching/. See Generation and KV cache.

Loading weights

graph LR
    Hub[Hugging Face Hub] -->|safetensors / pytorch_model.bin| Cache[Local cache ~/.cache/huggingface]
    Cache -->|load_state_dict| Loader[core_model_loading.py]
    Loader -->|WeightConverter ops| Model[PreTrainedModel]
    Loader -->|missing/unexpected| Report[loading_report.py]

V5 introduced a new weight-loading API (src/transformers/core_model_loading.py, 66K LOC) built around WeightConverter. Conversions can concatenate, split, transpose, and rename tensors during load, which is what enables clean integration with quantization and tensor parallelism. See from_pretrained and the V5 migration guide.

Cross-cutting subsystems

Subsystem	Source location	Purpose
Configuration	src/transformers/configuration_utils.py	Hyperparameter dataclass + JSON I/O
Modeling base	src/transformers/modeling_utils.py	PreTrainedModel, mixins, weight init
Generation	src/transformers/generation/	generate, logits processors, caches
Cache	src/transformers/cache_utils.py	KV cache abstractions (Dynamic, Static, Quantized, Sliding, Hybrid)
Attention dispatch	src/transformers/modeling_flash_attention_utils.py, integrations/sdpa_attention.py, integrations/flex_attention.py	Pluggable attention backends (SDPA, FA2/3/4, FlexAttention)
Tokenization	src/transformers/tokenization_*	Slow & fast tokenizers, mistral-common backend
Pipelines	src/transformers/pipelines/	25+ task pipelines
Trainer	src/transformers/trainer*.py	Full training loop
Quantization	src/transformers/quantizers/, integrations/	bitsandbytes, GPTQ, AWQ, mxfp4, FBGEMM, hqq, …
Integrations	src/transformers/integrations/	accelerate, deepspeed, peft, FSDP, TP, MoE
Hub I/O	src/transformers/utils/hub.py	Cached file resolution and uploads
CLI	src/transformers/cli/	chat, serve, download, add-new-model-like, env, version

Tests mirror the source

tests/ mirrors the src/transformers/ tree: each model directory has its own tests/models/<name>/test_modeling_<name>.py, common test mixins live at tests/test_modeling_common.py (291K LOC), and shared infrastructure is in tests/causal_lm_tester.py, tests/vlm_tester.py. Cross-cutting concerns get their own folders: tests/quantization/, tests/pipelines/, tests/generation/, tests/trainer/, tests/tensor_parallel/. See Testing.

CI and tooling

• Most CI runs on CircleCI (.circleci/) plus self-hosted GitHub Actions for GPU jobs (.github/workflows/self-scheduled*.yml, model_jobs.yml).
• make fix-repo (in Makefile) runs ruff, expands modular_*.py files into modeling_*.py, regenerates auto mappings, and updates docstrings.
• The repo-consistency surface is enforced by the scripts in utils/check_*.py (notably utils/check_repo.py, 66K LOC, and utils/modular_model_converter.py, 113K LOC).

See Tooling for details.