Lore

The story of how transformers evolved from a small BERT port into the model-definition framework for the open-source ML ecosystem.

Eras

The pytorch-pretrained-BERT era (Oct 2018 – Jul 2019)

The repository began as pytorch-pretrained-bert with the initial commit 43badf21 on 2018-10-29. The first releases shipped a single architecture (BERT) ported from the original Google TensorFlow code. Tag v0.5.0 was cut on 2019-07-16. The early codebase was tight: a few model files, basic tokenization, and a mission to make BERT usable in PyTorch.

Key events:

• 2018-10-29: Initial commit by Thomas Wolf and Anthony MOI.
• 2018-12: First multi-architecture release (v0.4.0); GPT, Transformer-XL, OpenAI GPT enter the repo.
• 2019-02: Python 2 dropped (PR #254).

The pytorch-transformers era (Jul 2019 – Sep 2019)

A short-lived rename to pytorch-transformers at v1.0.0 (2019-07) reflected the library's expansion beyond BERT. By this point GPT-2, RoBERTa, DistilBERT, and XLNet were in.

The 🤗 Transformers era (Sep 2019 – Nov 2020)

The library became transformers at v2.0.0 (2019-09-26), absorbing both PyTorch and TensorFlow 2 backends. This is the period where it became the de facto NLP toolkit. Tag v3.0.0 (2020-06-29) brought the Trainer API.

Defining changes of this era:

• Added Pipeline API (mid-2019).
• Added Trainer (src/transformers/trainer.py) in early 2020.
• Added from_pretrained/save_pretrained/push_to_hub as first-class concepts.

The vision-and-audio era (Nov 2020 – mid-2022)

v4.0.0 was cut on 2020-11-30. Over the next ~18 months the library expanded out of NLP. Vision Transformer (ViT), Wav2Vec2, CLIP, BEiT, Whisper-style audio models, and DETR landed. The number of model directories crossed 100, then 200.

Key shifts:

• Modality processors split into tokenizers, image processors, feature extractors, video processors, and unified Processor classes for multimodal models (src/transformers/processing_utils.py matured during this era).
• Half-precision and gradient checkpointing became standard Trainer knobs.

The integrations era (mid-2022 – Aug 2023)

Quantization, distributed training, and parameter-efficient fine-tuning entered as first-class citizens, but were initially scattered. PR #25599 on 2023-08-25 moved them under a single src/transformers/integrations/ folder, marked with the famous "🚨🚨🚨" prefix to flag breaking moves.

Key events:

• 2022: bitsandbytes and DeepSpeed integrations stabilize.
• 2023-08: integrations/ folder established.
• 2023-12: HfQuantizer introduced (PR #26610), separating quantization concerns from modeling_utils.py.
• 2023-12: New Cache abstraction introduced (PR #26681), pulling KV cache logic out of every modeling file into src/transformers/cache_utils.py.

The LLM-infra era (2024 – early 2025)

The community shifted to running ever-larger models on commodity hardware. The library responded with:

• 2024-06: Chat templates extended to function-calling and RAG (PR #30621). src/transformers/utils/chat_template_utils.py becomes a core file.
• 2024 throughout: 30+ quantization backends added (mxfp4, FBGEMM-FP8, hqq, quanto, torchao, VPTQ, SinQ, SpQR, …).
• 2025-03: Tensor-parallel refactor (PR #36539) rewrites src/transformers/integrations/tensor_parallel.py (now ~66K LOC) so models opt in via tp_plan.
• 2025-08: Continuous-batching refactor (PR #40426) introduces src/transformers/generation/continuous_batching/, enabling production-grade serving.

The v5 era (late 2025 – now)

v5.0 is the largest breaking release in the library's history. The work landed in 2025-2026 and includes:

• TensorFlow and JAX backends removed (PR #40760). PyTorch becomes the sole supported backend. See V5 migration.
• New weight-loading API built around WeightConverter (PR #41580). Replaces ad-hoc state-dict munging with composable conversion ops, enabling clean integration of quantization, tensor parallelism, and MoE sharding.
• Tokenizer rewrite with simpler TokenizersBackend and mistral-common support.
• CLI migrated to Typer on 2025-10-16 (PR #41487). transformers chat and transformers serve mature into OpenAI-compatible endpoints.
• The rename PretrainedConfig → PreTrainedConfig (with alias preserved).

The current __version__ in src/transformers/__init__.py is 5.8.0.dev0. The latest tagged release at the time this wiki was generated is v5.7.0.

Longest-standing features

Component	First appeared	Still active
BERT modeling	2018-10 (initial commit)	Yes; src/transformers/models/bert/
Tokenizer base class	2018-10	Yes; rewritten several times in src/transformers/tokenization_utils_base.py
from_pretrained / save_pretrained	2018-12 (v0.4.0)	Yes; the canonical Hub I/O API
Trainer	2020 (during v3)	Yes; src/transformers/trainer.py
Pipeline	2019 (during v2)	Yes; src/transformers/pipelines/
# Copied from mechanism	2020	Yes; complemented by modular_*.py since 2024

Deprecated and removed features

• TensorFlow 2 backend — present from v2 (Sep 2019) through v4. Removed in v5 (PR #40760, 2025).
• Flax/JAX backend — present from v4. Removed in v5 (PR #40760, 2025).
• pytorch_model.bin — superseded by safetensors as the default checkpoint format.
• Trainer for TF/Flax (TFTrainer, FlaxTrainer) — removed alongside the backends.
• Several research-only models were moved to examples/research_projects/ over the years (mmbt, bertology, etc.).
• TF/Flax CI matrices and Dockerfiles — culled from docker/ and .github/workflows/ during v5 prep.

Major rewrites

• Cache abstraction (Dec 2023, PR #26681). Removed cache code from every modeling file.
• Quantization decoupling (Jan 2024, PR #26610). Introduced HfQuantizer and the quantizers/ folder.
• Integrations folder (Aug 2023, PR #25599). Moved accelerate, peft, deepspeed, bitsandbytes into integrations/.
• Tensor parallelism (Mar 2025, PR #36539). New tp_plan API and centralized TP logic.
• Continuous batching (Aug 2025, PR #40426). Production-grade serving support.
• Weight loading (Apr 2025+, PR #41580). WeightConverter API replaces _load_pretrained_model plumbing.
• CLI to Typer (Oct 2025, PR #41487). transformers CLI rewritten with Typer.
• TF/Flax removal (2025, PR #40760).

Growth trajectory

• 2018-10 — 1 model (BERT), ~5K LOC.
• 2019-09 — ~10 models, multi-backend (PyTorch + TF).
• 2020-11 — ~50 models, Trainer mature, Pipeline mature, vision models begin.
• 2023-08 — ~250 models, integrations/ folder created, quantization landed.
• 2024 throughout — quantization backend explosion, chat templates expand.
• 2025 — 400+ models, tensor parallel + continuous batching + new weight-loading API.
• 2026-04 — 462 model directories, ~149K library LOC, ~412K test LOC, 22,758 commits.