Modeling

Purpose

PreTrainedModel is the base class every concrete model inherits from. It supplies from_pretrained, save_pretrained, push_to_hub, weight tying, gradient checkpointing, attention dispatch, FSDP/TP integration, quantization hooks, and the universal generate mixin. It is the longest source file in the library at 5,042 lines.

Key abstractions

Class / function	File	Role
PreTrainedModel	src/transformers/modeling_utils.py	Base class for all models
GenerationMixin	src/transformers/generation/utils.py	Adds generate() to causal/seq2seq models
<Arch>PreTrainedModel	src/transformers/models/<arch>/modeling_<arch>.py	Per-architecture abstract base
Output dataclasses	src/transformers/modeling_outputs.py (108K LOC)	BaseModelOutput, CausalLMOutputWithPast, …
ModuleUtilsMixin	src/transformers/modeling_utils.py	Helpers for parameter dtype, device, count
_LazyModule	src/transformers/utils/import_utils.py	Lazy import dispatch
Layer mixins	src/transformers/modeling_layers.py	Reusable layer scaffolding
Weight init	src/transformers/initialization.py	Standard init schemes

What PreTrainedModel does

• Hub I/O. from_pretrained, save_pretrained, push_to_hub — see from_pretrained.
• Weight tying. tie_weights() ties input and output embeddings when configured.
• Gradient checkpointing. gradient_checkpointing_enable/disable() and per-layer hooks.
• Attention dispatch. Sets _attn_implementation and routes models to SDPA / FA2 / FA3 / FA4 / FlexAttention / paged variants. See Attention.
• Quantization hooks. Lifecycle calls into HfQuantizer (see Quantization).
• FSDP and TP. _init_weights, _no_split_modules, _keep_in_fp32_modules, _tp_plan, _fsdp_pretrained_module_class. See Integrations.
• Output bookkeeping. Dataclass outputs from modeling_outputs.py so Trainer and Pipeline can introspect predictions, hidden states, and attentions.
• from_config for instantiating without loading weights.

Where forward lives

The actual forward of a model is in its per-architecture modeling_<name>.py. The "One Model, One File" tenet forbids hiding it behind a base class. PreTrainedModel does not implement forward; it provides the scaffolding around which forward is written.

A typical LlamaForCausalLM.forward (in src/transformers/models/llama/modeling_llama.py) does:

1. Pass inputs through embedding + decoder layers.
2. Read/write past_key_values from a Cache.
3. Compute logits via lm_head.
4. If labels are provided, compute cross-entropy loss.
5. Return a CausalLMOutputWithPast dataclass.

The lifecycle of from_pretrained

sequenceDiagram
    participant U as User
    participant FP as from_pretrained
    participant Hub as Hub cache (utils/hub.py)
    participant CFG as PretrainedConfig
    participant LD as core_model_loading.py
    participant M as Model

    U->>FP: AutoModelForCausalLM.from_pretrained(repo_id)
    FP->>Hub: resolve config.json + safetensors shards
    Hub-->>FP: local paths
    FP->>CFG: from_dict(json)
    FP->>FP: pick concrete class via CONFIG_MAPPING / MODEL_FOR_CAUSAL_LM_MAPPING
    FP->>M: cls(config) — meta tensors
    FP->>LD: load_state_dict_with_conversions
    LD->>LD: apply WeightConverter ops (concat, transpose, rename)
    LD-->>M: parameters materialized
    FP->>M: tie_weights()
    FP->>M: post_init() — _init_weights for missing keys
    FP-->>U: model (eval mode)

core_model_loading.py (66K LOC) is the V5 weight-loading engine. It sequences tensor materialization to minimize peak memory and supports complex transformations such as concatenating Q/K/V projections during load. See from_pretrained.

Mixins that extend PreTrainedModel

• GenerationMixin (src/transformers/generation/utils.py) adds generate. Most decoder-style models inherit from both PreTrainedModel and GenerationMixin.
• BackboneMixin (src/transformers/backbone_utils.py) adds the out_indices / feature pyramid contract used by detection and segmentation models.
• BackboneOutputs and friends in src/transformers/modeling_outputs.py.

Training-aware features

• Gradient checkpointing is configured per-model via _supports_gradient_checkpointing and per-layer hooks. Used aggressively by Trainer.
• Mixed precision is honoured by the dtype handling in _init_weights and _keep_in_fp32_modules (some operations remain in FP32 for stability).
• torch.compile is supported when the model declares _supports_static_cache = True and uses a StaticCache from cache_utils.py.

Integration points

• Pipelines call model.forward (and generate for generative tasks).
• Trainer calls model.forward inside its training loop and uses model.gradient_checkpointing_enable(), model.tie_weights(), etc.
• Quantization replaces linear layers via the HfQuantizer API during from_pretrained.
• Tensor parallelism shards the modules according to model._tp_plan.

Entry points for modification

• To add a new architecture, write modeling_<name>.py (or a modular_<name>.py shard) that defines <Arch>Model, <Arch>PreTrainedModel, and the head classes. Register them in auto_mappings.py. See Modular models.
• To change cross-cutting behaviour (e.g., add a new attention backend), edit src/transformers/modeling_utils.py and the relevant integration file in src/transformers/integrations/. Tests live in tests/test_modeling_common.py.
• To affect only a subset of models, override the relevant method on the per-arch <Arch>PreTrainedModel.