Patterns and conventions

The conventions that govern day-to-day code in transformers. These are the rules that come up most often in PR reviews.

One Model, One File

The library's flagship tenet. Every architecture's core inference and training logic must live in a single readable file (modeling_<name>.py), even at the cost of code duplication. The team's reasoning, captured in docs/source/en/philosophy.md:

Code is the Product. Optimize for reading and diff-ing. Prefer explicit names over clever indirection.

In practice that means a contributor reading src/transformers/models/llama/modeling_llama.py should never have to chase imports through five other modeling files to understand the forward pass.

# Copied from ...

The older mechanism for keeping shared code in sync without abstracting it.

# Copied from transformers.models.llama.modeling_llama.LlamaRMSNorm
class MistralRMSNorm(nn.Module):
    def __init__(self, hidden_size, eps=1e-6):
        ...

make fix-repo invokes utils/check_copies.py (46K LOC) and rewrites every # Copied from block from the source. Editing inside such a block is futile — the next make fix-repo will revert it. Workflow:

• To change the shared behaviour, edit the source class and let make fix-repo propagate.
• To diverge, delete the # Copied from ... comment and add a one-line explanation of why this copy intentionally drifts.

The directive supports tweaks like # Copied from … with Llama->Mistral, llama->mistral to perform automated renames during the copy.

Modular models (the newer way)

Adding a new architecture by writing a modular_<name>.py shard is now the preferred path. The shard imports from existing models and overrides what differs. make fix-repo then expands it into the user-visible modeling_<name>.py, configuration_<name>.py, tokenizer files, and image-processor files.

Example skeleton (src/transformers/models/<arch>/modular_<arch>.py):

from ..llama.modeling_llama import LlamaForCausalLM
from ..llama.configuration_llama import LlamaConfig

class MyArchConfig(LlamaConfig):
    model_type = "my_arch"

class MyArchForCausalLM(LlamaForCausalLM):
    pass

The expansion engine is utils/modular_model_converter.py (113K LOC). It performs class-renaming, attribute overrides, dependency tracking, and preserves the "One Model, One File" outcome for end users while giving maintainers a small diff.

Reference: docs/source/en/modular_transformers.md.

230 of the 462 model directories now ship a modular shard.

Naming

Pattern	Example	Meaning
<Arch>Config	LlamaConfig	Configuration class
<Arch>Model	LlamaModel	Backbone (no head)
<Arch>For<Task>	LlamaForCausalLM, BertForSequenceClassification	Task-specific head wrapper
<Arch>PreTrainedModel	LlamaPreTrainedModel	Per-model abstract base
<Arch>Tokenizer, <Arch>TokenizerFast	LlamaTokenizerFast	Tokenizer variants
<Arch>ImageProcessor, <Arch>VideoProcessor, <Arch>FeatureExtractor, <Arch>Processor	LlavaProcessor	Preprocessors

Auto classes are AutoConfig, AutoModel, AutoModelFor<Task>, AutoTokenizer, AutoImageProcessor, AutoVideoProcessor, AutoFeatureExtractor, AutoProcessor. See Auto classes.

Attention dispatch

Models declare an attention factory rather than coding attention inline. The relevant pieces:

• src/transformers/modeling_flash_attention_utils.py — FA2/3/4 dispatch.
• src/transformers/integrations/sdpa_attention.py — SDPA dispatch.
• src/transformers/integrations/flex_attention.py — FlexAttention dispatch.
• src/transformers/integrations/eager_paged.py, flash_paged.py, sdpa_paged.py — paged variants for continuous batching.
• _attn_implementation selects the backend at runtime; users override via model = AutoModel.from_pretrained(..., attn_implementation="flash_attention_2").

Models that gate attention behind if/elif per backend are reviewed against the dispatch table; new backends should be added to the dispatcher, not duplicated in every model.

Cache abstractions

Direct manipulation of past_key_values tuples is deprecated. Models that need stateful KV caching use Cache subclasses from src/transformers/cache_utils.py:

• DynamicCache — grows on the fly (default for greedy/sample).
• StaticCache — fixed shape; required for torch.compile and serving.
• SlidingWindowCache, HybridCache — for sliding-window or hybrid (Mamba-like) architectures.
• EncoderDecoderCache — encoder-decoder pairs.
• QuantizedCache, OffloadedCache — memory optimizations.

See Cache.

Output dataclasses

Every model returns a dataclass from src/transformers/modeling_outputs.py (108K LOC) — never a raw tuple. Common ones:

• BaseModelOutput, BaseModelOutputWithPast, BaseModelOutputWithPooling.
• CausalLMOutput, CausalLMOutputWithPast, Seq2SeqLMOutput.
• ImageClassifierOutput, ObjectDetectionOutput, SemanticSegmenterOutput.

Custom output dataclasses go in the per-model file but must subclass an existing base class so Trainer and Pipeline can introspect the fields.

Auto-docstring

Docstrings on __init__, forward, and from_pretrained are auto-checked and largely auto-generated. The engine is src/transformers/utils/auto_docstring.py (165K LOC). Required behaviour:

• Use the @auto_docstring decorator (or rely on make fix-repo to add it).
• Document arguments by name only; the rendering pulls types and defaults from the signature.
• The reference doc is docs/source/en/auto_docstring.md.

The check that enforces this is utils/check_docstrings.py.

Dummy objects

Optional dependencies are gated by is_<dep>_available() calls (e.g., is_torch_available, is_vision_available). When the dep is missing, importing a class still works because src/transformers/utils/dummy_*.py defines stubs that raise informative errors on first use. The dummies are auto-generated by utils/check_dummies.py.

Auto mappings must be sorted

src/transformers/models/auto/auto_mappings.py is hand-edited but the OrderedDicts must be sorted by utils/sort_auto_mappings.py (which make fix-repo runs). Forgetting this fails CI.

Don't add new dependencies casually

Adding a top-level dependency requires:

1. Editing setup.py _deps (with a version pin).
2. Running make fix-repo to regenerate src/transformers/dependency_versions_table.py.
3. Justifying it in the PR description; the team aggressively pushes back on new mandatory deps.

Optional deps go in extras and use the dummy-object pattern.

Don't break backward compatibility silently

The "Backwards Compatibility" tenet means:

• Public surfaces (from transformers import X) should not change.
• from_pretrained must keep working with old Hub artifacts.
• Old config keys are migrated, not removed (PretrainedConfig._handle_deprecated_kwargs).

Breaking changes go through the v-bump cycle (v5.0 is the recent example). PRs proposing breakage need core-maintainer approval.