Glossary

Project-specific vocabulary you will encounter when reading the codebase or PRs.

Core architecture terms

• Model triplet — every architecture's three required classes: configuration, model, preprocessor. Defined in src/transformers/configuration_utils.py, src/transformers/modeling_utils.py, and a tokenizer/processor file.
• PreTrainedModel — the base class for all model classes. Adds from_pretrained, save_pretrained, push_to_hub, weight tying, gradient checkpointing, attention dispatch, FSDP/TP integration, and quantization hooks. Lives in src/transformers/modeling_utils.py (~5K LOC).
• PreTrainedConfig / PretrainedConfig — the config dataclass base. The two names exist because v5 renamed it but kept the alias for backward compatibility (src/transformers/configuration_utils.py).
• Model head — a model class that adds a task-specific output layer. E.g., LlamaForCausalLM, BertForSequenceClassification, ViTForImageClassification. The naming pattern <Arch>For<Task> is universal.
• Auto class — late-binding factory like AutoModelForCausalLM. Looks up model_type in the config to find the concrete class. Source: src/transformers/models/auto/.

Code organization

• One Model, One File — design tenet: every model's core logic lives in a single modeling_<name>.py file readable top-to-bottom. Yes, this duplicates code. The team accepts that cost so contributors and users can patch a single file. See docs/source/en/philosophy.md.
• # Copied from ... — a marker comment that links a class/function to its origin. make fix-repo keeps the copy in sync with the original. Editing inside a # Copied from block is futile because the block will be regenerated.
• Modular — newer alternative to # Copied from. A contributor writes a small modular_<name>.py shard that inherits from another model; make fix-repo (specifically utils/modular_model_converter.py, 113K LOC) expands it into the full modeling_<name>.py, configuration_<name>.py, etc. The expanded files are the ones users read; the modular shard is what maintainers review. There are 230 modular_*.py shards in the tree today. See Modular models.
• add-new-model-like — a CLI command (src/transformers/cli/add_new_model_like.py) that scaffolds a new model directory by copying and renaming an existing one.

Generation and caching

• generate — the universal autoregressive decoding method, defined in src/transformers/generation/utils.py (3,887 LOC) and mixed into models via GenerationMixin. Supports greedy, beam, contrastive, sample, assisted, DoLa.
• Logits processor — a callable that mutates next-token logits before sampling (temperature, top-k, top-p, repetition penalty, JSON-schema-guided decoding, etc.). See src/transformers/generation/logits_process.py (150K LOC).
• Stopping criterion — a callable that can halt generation (max length, EOS, stop strings). See src/transformers/generation/stopping_criteria.py.
• Streamer — pushes generated tokens to a queue or stdout as they are produced. See src/transformers/generation/streamers.py.
• KV cache — the past key/value tensors of the attention layers. Reusing them across decoding steps is what makes autoregressive generation fast. The cache hierarchy lives in src/transformers/cache_utils.py (1,574 LOC) and includes DynamicCache, StaticCache, SlidingWindowCache, HybridCache, EncoderDecoderCache, QuantizedCache, OffloadedCache.
• Continuous batching — server-side scheduling that mixes prefill and decode steps from different requests in the same batch using paged attention. See src/transformers/generation/continuous_batching/.

Attention backends

• SDPA — torch.nn.functional.scaled_dot_product_attention. The default backend on PyTorch 2+.
• FlashAttention 2 / 3 / 4 — pluggable backends from the flash-attn package. Routed via src/transformers/modeling_flash_attention_utils.py.
• FlexAttention — torch.nn.attention.flex_attention. Routed via src/transformers/integrations/flex_attention.py.
• Eager — the manual softmax(QK^T/sqrt(d)) V implementation, used as fallback or when others are unavailable.
• Paged / paged kernels — block-allocated KV cache used by continuous batching (src/transformers/integrations/flash_paged.py, eager_paged.py, sdpa_paged.py).

Tokenization

• Slow tokenizer — pure-Python tokenizer derived from PreTrainedTokenizer. Source: src/transformers/tokenization_python.py.
• Fast tokenizer — Rust-backed PreTrainedTokenizerFast, wrapping the tokenizers library. Source: src/transformers/tokenization_utils_tokenizers.py (66K LOC).
• mistral-common — third tokenizer backend introduced by Mistral. See src/transformers/tokenization_mistral_common.py.
• Chat template — Jinja2 template stored on the tokenizer that renders a message list into the model's expected prompt format. See src/transformers/utils/chat_template_utils.py and Chat templates.

Training and quantization

• Trainer — the all-in-one training loop class (src/transformers/trainer.py, 4,418 LOC).
• TrainingArguments — the dataclass holding every training knob (src/transformers/training_args.py, 2,868 LOC).
• PEFT — Parameter-Efficient Fine-Tuning. Integrated via src/transformers/integrations/peft.py (53K LOC). LoRA, prefix-tuning, etc. are loaded as adapters on top of a base model.
• bitsandbytes — popular int8/4-bit quantizer. Adapter at src/transformers/integrations/bitsandbytes.py.
• GPTQ / AWQ / mxfp4 / FBGEMM-FP8 / hqq / quanto / torchao / VPTQ / SinQ / SpQR / Higgs / FineGrainedFP8 / EETQ / AQLM / Quark / FPQuant — supported quantization backends, each with a quantizer class in src/transformers/quantizers/ and an integration helper in src/transformers/integrations/.
• Tensor parallel (TP) — sharding model weights across GPUs along a tensor dimension. Implemented at src/transformers/integrations/tensor_parallel.py (66K LOC). Triggered with tp_plan="auto" in from_pretrained.
• Expert parallel (EP) — sharding MoE experts across devices. See src/transformers/integrations/moe.py.

Data

• Data collator — a callable that turns a list of examples into a padded batch. See src/transformers/data/data_collator.py. Common variants: DataCollatorWithPadding, DataCollatorForLanguageModeling, DataCollatorForSeq2Seq, DataCollatorForTokenClassification, DataCollatorWithFlattening.
• Processor (multimodal) — a class that bundles a tokenizer + image/feature/video processor for multimodal models. See src/transformers/processing_utils.py (101K LOC).

Repo workflow

• make fix-repo — runs every auto-fixer: ruff format, modular expansion, copies, doc TOC, docstrings. Mandatory before PR.
• make check-repo — read-only consistency check that mirrors what CI runs.
• CircleCI — primary CI provider for PR checks (.circleci/config.yml).
• Self-hosted runners — used for GPU/AMD/Intel/AMD CI under .github/workflows/self-scheduled*.yml.
• Tiny models — small dummy versions of every architecture, used in fast tests; created by utils/create_dummy_models.py (84K LOC).
• tests_fetcher.py — utils/tests_fetcher.py (52K LOC) computes which tests cover the changed files in a branch.