Text models

The original constituency of the library. Text models are by far the largest single group and span encoder-only (BERT family), decoder-only (GPT/Llama family), and encoder-decoder (T5/BART family) architectures. New decoder LMs continue to land at a fast pace.

Decoder LMs (causal language modeling)

The dominant family of recent additions. Each typically declares <Arch>ForCausalLM and exposes tp_plan for tensor parallelism.

Family	Representative directories	Notes
GPT-2 family	gpt2/, gpt_neo/, gpt_neox/, gpt_neox_japanese/, gpt_sw3/, gptj/, gpt_bigcode/, gpt_oss/, openai/	Pre-Llama era; many still actively used
Llama family	llama/, llama4/, code_llama/	Reference architecture for tp_plan
Mistral family	mistral/, mistral3/, mistral4/, ministral/, ministral3/, mixtral/, minimax/, minimax_m2/	Sliding-window cache examples
Qwen family	qwen2/, qwen2_moe/, qwen3/, qwen3_5/, qwen3_5_moe/, qwen3_moe/, qwen3_next/	Many MoE variants
Gemma family	gemma/, gemma2/, gemma3/, gemma3n/, gemma4/, recurrent_gemma/, t5gemma/, t5gemma2/, vaultgemma/	Hybrid cache examples
Phi family	phi/, phi3/, phimoe/	Microsoft research
Cohere	cohere/, cohere2/	Command R
DeepSeek	deepseek_v2/, deepseek_v3/	MLA attention, MoE
GLM	glm/, glm4/, glm4_moe/, glm4_moe_lite/, glm_moe_dsa/	THUDM family
Falcon	falcon/, falcon_h1/, falcon_mamba/	Mix of standard and SSM hybrids
OLMo	olmo/, olmo2/, olmo3/, olmoe/, olmo_hybrid/, flex_olmo/	Allen AI fully-open models
Granite	granite/, granitemoe/, granitemoehybrid/, granitemoeshared/	IBM
Xlstm/Mamba/Hybrid SSM	mamba/, mamba2/, falcon_mamba/, bamba/, nemotron_h/, xlstm/, zamba/, zamba2/, jamba/, jetmoe/	State-space and hybrid SSM/attention
Other recent	apertus/, arcee/, bitnet/, dbrx/, doge/, dots1/, ernie4_5/, ernie4_5_moe/, exaone4/, exaone_moe/, helium/, hunyuan_v1_dense/, hunyuan_v1_moe/, hy_v3/, lfm2/, lfm2_moe/, longcat_flash/, nanochat/, nemotron/, persimmon/, seed_oss/, smollm3/, solar_open/, stablelm/, starcoder2/, cwm/, laguna/, jais2/	The pace is roughly one a week

Encoder-only (masked LM, classification, NER)

Family	Directories	Notes
BERT family	bert/, bert_generation/, bert_japanese/, bertweet/, roberta/, roberta_prelayernorm/, xlm_roberta/, xlm_roberta_xl/, camembert/, flaubert/, xlm/, herbert/, phobert/, roformer/, roc_bert/	The classics
Distilled	distilbert/	Smaller, faster
Long context	bigbird/ (big_bird/), bigbird_pegasus/, longformer/, nystromformer/, reformer/, xmod/, yoso/	Sparse / linear attention
Modern	modernbert/, modernbert_decoder/, modernvbert/, nomic_bert/, eurobert/	Post-2024 efficient encoders
Token / token-pair	electra/, convbert/, funnel/, mpnet/, splinter/, squeezebert/, mobilebert/, mobilevitv2/, mra/, lilt/, luke/, mluke/, markuplm/, tapas/, byt5/	Specialized tasks
Embeddings	jina_embeddings_v3/	Sentence/embedding model
Domain-specific	biogpt/, esm/, cwm/, udop/, markuplm/, pi0/, paddleocr_vl/	Biology, code, document AI
Privacy filter	openai_privacy_filter/	Content moderation classifier
Other	albert/, xlnet/, canine/, deberta/, deberta_v2/, electra/, ernie/, fnet/, ibert/, rembert/, bros/, myt5/, mgp_str/

Encoder-decoder (seq2seq)

Family	Directories	Notes
T5 family	t5/, mt5/, umt5/, byt5/, longt5/, t5gemma/, t5gemma2/	Span-based pretraining
BART family	bart/, mbart/, mbart50/, barthez/, bartpho/, mvp/, plbart/	Denoising pretraining
Translation specialists	m2m_100/, marian/, nllb/, nllb_moe/, seamless_m4t/, seamless_m4t_v2/, pegasus/, pegasus_x/, prophetnet/
Generic wrappers	encoder_decoder/, vision_encoder_decoder/, speech_encoder_decoder/, vision_text_dual_encoder/	Compose any encoder with any decoder
Other	blenderbot/, blenderbot_small/, dialogpt/, fsmt/, led/, switch_transformers/ (sparse MoE), mass-style models

Specialty text architectures

Directory	Purpose
rag/	Retrieval-augmented generation orchestration
tapas/	Table understanding
markuplm/, udop/, layoutlm/, layoutlmv2/, layoutlmv3/, layoutxlm/, donut/, nougat/	Document AI (overlap with multimodal)
cpm/, cpmant/	Chinese pre-trained models
decision_transformer/	Offline RL via transformer
time_series_transformer/, informer/, autoformer/, patchtst/, patchtsmixer/, timesfm/, timesfm2_5/	Time-series (encoder-decoder for forecasting)

Reading guidance

For each family, the canonical file is src/transformers/models/<family>/modeling_<family>.py. Most also have:

• A tokenization_<family>.py (slow) and tokenization_<family>_fast.py (fast).
• A user-facing doc page at docs/source/en/model_doc/<family>.md.

For modular shards, prefer reading modular_<family>.py first — it shows what the architecture inherits from and what it overrides, which is often a tighter mental model than the expanded modeling file.

See also

• Auto classes — how AutoModelForCausalLM, AutoModelForSeq2SeqLM, etc. resolve to the right class.
• Modular models — the shard mechanism most new text models use.
• Tensor parallelism — the _tp_plan per-model declarations.
• Generation — what generate does for these models.