Integrations

Purpose

src/transformers/integrations/ contains adapters that bridge transformers to third-party libraries: distributed training (accelerate, deepspeed), parameter-efficient fine-tuning (peft), distributed parallelism (FSDP, tensor parallel, expert parallel), attention backends, quantization runtimes, custom kernels, and experiment trackers.

Top contents

File	Purpose
__init__.py	Stable re-exports
accelerate.py (41K LOC)	AcceleratorConfig and Trainer ↔ accelerate glue
deepspeed.py (33K LOC)	DeepSpeed ZeRO config + initialization
peft.py (53K LOC)	PEFT adapters, LoRA loading
fsdp.py	FSDP wrapping and config
tensor_parallel.py (66K LOC)	Tensor parallel sharding
moe.py (25K LOC)	Mixture-of-Experts dispatcher and expert parallel
flash_attention.py, sdpa_attention.py, flex_attention.py, eager_paged.py, flash_paged.py, sdpa_paged.py, npu_flash_attention.py	Attention backends
bitsandbytes.py, awq.py, eetq.py, hqq.py, quanto.py, torchao.py, mxfp4.py, fbgemm_fp8.py, finegrained_fp8.py, bitnet.py, aqlm.py, vptq.py, sinq.py, spqr.py, higgs.py, quark.py, fp_quant.py, fouroversix.py, sonicmoe.py, liger.py, mistral.py, metal_quantization.py, executorch.py, ggml.py, tiktoken.py, tpu.py, neftune.py, hub_kernels.py	Per-backend adapters
integration_utils.py (118K LOC)	Experiment tracker callbacks (W&B, MLFlow, Tensorboard, Comet, ClearML, DVCLive, SwanLab, AzureML, Neptune)

What lives where

Distributed training

• accelerate.py exposes AcceleratorConfig, the dataclass that mirrors accelerate.Accelerator settings consumed by Trainer. It builds Accelerator and wires hooks for gradient accumulation, mixed precision, and gradient clipping.
• deepspeed.py initializes DeepSpeed ZeRO from a JSON config or the inline subset accepted by TrainingArguments.deepspeed. Implements stage-3 weight loading helpers that interoperate with from_pretrained.
• fsdp.py builds the FSDP wrapper and applies auto_wrap_policy based on model._fsdp_pretrained_module_class.

Parallelism

• tensor_parallel.py (66K LOC) implements row/column-parallel sharding, all-gather/all-reduce hooks, and the tp_plan mechanism. Models opt in by declaring _tp_plan (a dict mapping module names → colwise/rowwise/gather/replicate). See Tensor parallelism.
• moe.py (25K LOC) handles MoE expert dispatch and expert parallelism. Combined with tensor_parallel.py for full 3D parallelism.

Attention backends

See Attention. The integrations directory hosts the SDPA, FlexAttention, paged variants, and NPU flash attention. FA2/3/4 entry points live in src/transformers/modeling_flash_attention_utils.py outside this folder for historical reasons but the dispatcher unifies them.

PEFT

peft.py (53K LOC) is the integration with peft. It teaches PreTrainedModel to:

• Load LoRA / IA³ / prefix-tuning adapters with model.load_adapter("repo").
• Combine multiple adapters with model.set_adapter("name").
• Save adapters with model.save_pretrained (writing only the small adapter weights).
• Train adapters via Trainer (which integrates with peft.PeftModel).

It also handles the quantized + LoRA combo (QLoRA) for bitsandbytes, hqq, quanto, torchao.

Quantization

Each quantization backend has both an integration file (which performs layer replacement and kernel calls) and a quantizer file under src/transformers/quantizers/. See Quantization.

Custom kernels

hub_kernels.py (19K LOC) integrates with the kernels project. Model files can declare a kernel via the @use_kernel_forward_from_hub("repo_id") decorator; at import time the kernel is fetched and substituted for the eager PyTorch implementation. Used for fused RMSNorm, fused MoE, etc.

liger.py integrates with Liger Kernel (Triton-based fused kernels for RMSNorm, RoPE, SwiGLU, cross-entropy, …) optionally enabled by Trainer(args=TrainingArguments(use_liger_kernel=True)).

Experiment trackers

integration_utils.py registers callbacks for every supported tracker. Activated automatically when the backend is installed; controlled via TrainingArguments.report_to.

Special-purpose

• executorch.py (49K LOC) exports models to PyTorch ExecuTorch for on-device inference.
• ggml.py (33K LOC) reads/writes GGML/GGUF files for llama.cpp interop.
• tiktoken.py glue for OpenAI's BPE tokenizer.
• mistral.py glue for the mistral-common tokenizer/inference utilities.
• tpu.py Cloud TPU support helpers.
• neftune.py NEFTune noisy-embedding fine-tuning.
• metal_quantization.py Apple Silicon Metal support.
• sonicmoe.py SonicMoE dispatcher (used by AudioFlamingo and friends).

Loading sequence

graph LR
    FP[from_pretrained] --> Q{Quantization config?}
    Q -->|yes| Quant[HfQuantizer.process_before]
    Q -->|no| Skip
    Quant --> Load[Load weights]
    Skip --> Load
    Load --> TP{tp_plan / FSDP?}
    TP -->|yes| Shard[tensor_parallel.shard / fsdp.wrap]
    TP -->|no| Done
    Shard --> Done

Integration points

• Trainer — accelerate, deepspeed, peft, FSDP, TP, integration_utils.
• Modeling — attention dispatch, kernels, quantization hooks.
• Quantization — per-backend integration helpers.
• Tensor parallelism — tensor_parallel.py.

Entry points for modification

• New tracker → add a callback class to integration_utils.py and register it in the tracker dispatch table.
• New attention backend → add a file here, update the dispatcher in src/transformers/modeling_utils.py and modeling_flash_attention_utils.py.
• New parallelism strategy → add a file here, update tensor_parallel.py or moe.py to expose the new strategy through tp_plan / ep_plan.
• New kernel → register via hub_kernels.py or contribute to the kernels-community repo and add the @use_kernel_forward_from_hub decorator on the relevant model layer.