from_pretrained

The universal load path. Every config, model, tokenizer, image processor, video processor, feature extractor, and multimodal processor in the library exposes the same from_pretrained(repo_id_or_path, **kwargs) and save_pretrained(directory) methods, plus push_to_hub(repo_id).

Where the work happens

Concern	File
Hub cache resolution and download	src/transformers/utils/hub.py (39K LOC)
Model weight loading and conversion	src/transformers/core_model_loading.py (66K LOC)
Conversion mapping per architecture	src/transformers/conversion_mapping.py (36K LOC)
Loading reports	src/transformers/utils/loading_report.py
Quantization lifecycle	src/transformers/quantizers/base.py
Tensor parallel sharding	src/transformers/integrations/tensor_parallel.py

The V5 weight-loading API

MIGRATION_GUIDE_V5.md explains the new WeightConverter API introduced in PR #41580. Operations are first-class:

class WeightConverter(WeightTransform):
    operations: list[ConversionOps]
    source_keys: Union[str, list[str]]
    target_keys: Union[str, list[str]]

A typical conversion fuses Q/K/V projections into a single qkv_proj:

WeightConverter(
    ["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"],
    "self_attn.qkv_proj",
    operations=[Concatenate(dim=0)],
)

The benefits the team called out in the migration guide:

• Cleaner expression of checkpoint transformations.
• Reversible (load+save round-trips reproduce the same checkpoint).
• Faster loading via scheduled tensor materialization.
• Composable with quantization, tensor parallelism, and MoE sharding without bespoke code per case.

Typical sequence

sequenceDiagram
    participant U as User
    participant FP as cls.from_pretrained
    participant Hub as utils/hub.py
    participant Cfg as PretrainedConfig
    participant Cls as Concrete model
    participant Loader as core_model_loading
    participant Q as HfQuantizer (optional)

    U->>FP: from_pretrained("repo_id", **kwargs)
    FP->>Hub: cached_file("config.json")
    Hub-->>FP: local path
    FP->>Cfg: from_dict(json)
    FP->>FP: kwargs override (dtype, device_map, attn_implementation, …)
    FP->>Cls: cls(config) on meta device
    alt quantization_config given
        FP->>Q: validate_environment + process_before_loading
    end
    FP->>Hub: cached_file("model.safetensors.index.json")
    FP->>Loader: load + apply WeightConverter ops + dispatch to devices
    Loader->>Cls: assign tensors
    alt tp_plan / fsdp
        Loader->>Cls: shard / wrap
    end
    Loader-->>FP: loading report (missing, unexpected, mismatches)
    FP->>Cls: post_init() + tie_weights
    alt quantization_config given
        FP->>Q: process_after_loading
    end
    FP-->>U: model

Important kwargs

| kwarg | Effect | | --------------------------------------------- | ----------------------------------------------------- | ---------------- | -------- | ----------------------- | | dtype=torch.bfloat16 | Cast parameters to bf16 (was torch_dtype pre-V5). | | device_map="auto" | Use accelerate to spread across devices. | | device_map={"": "cuda:0"} | Place all on cuda:0. | | attn_implementation="sdpa" | "flash_attention_2" | "flex_attention" | "eager" | Pick attention backend. | | quantization_config=BitsAndBytesConfig(...) | Activate quantization at load time. | | tp_plan="auto" | Tensor-parallel sharding (requires WORLD_SIZE > 1). | | low_cpu_mem_usage=True | Skip the meta-device → CPU staging. | | subfolder="checkpoint-1000" | Load from a sub-path of the repo. | | revision="abc123" | Specific commit / branch / tag. | | trust_remote_code=True | Allow auto_map modules from the Hub. | | use_safetensors=True/False | Force a serialization backend. |

save_pretrained

save_pretrained(directory) writes:

• config.json
• generation_config.json (if a GenerationConfig was set)
• model.safetensors (or sharded model-00001-of-N.safetensors + index)
• Tokenizer or processor files (when called on those classes)
• preprocessor_config.json for image/feature/video processors

For sharded checkpoints, the file model.safetensors.index.json maps tensor names to shards. The shard threshold is configurable.

push_to_hub

push_to_hub(repo_id) uploads the directory to the Hub. Implemented in src/transformers/utils/hub.py and the per-class PushToHubMixin. Optional kwargs:

• commit_message, commit_description.
• private=True for private repos.
• create_pr=True to push a draft PR.
• safe_serialization=True/False.

Cache layout

Local cache is at $HF_HOME (defaults to ~/.cache/huggingface). Files are stored under hub/models--<org>--<repo>/snapshots/<commit>/... with symlinks from refs/<branch> and the blobs/ folder. The cache uses filelock for safe concurrent access.

Environment variables:

• HF_HOME — root cache directory.
• HF_HUB_OFFLINE=1 — refuse to make network calls.
• TRANSFORMERS_OFFLINE=1 — same, scoped to this library.
• HF_HUB_DOWNLOAD_TIMEOUT — download HTTP timeout (defaults to 60 in this repo's pytest env, see pyproject.toml).

Loading reports

from_pretrained no longer prints free-form warnings. It assembles a structured report (src/transformers/utils/loading_report.py) with:

• Successfully loaded keys.
• Missing keys (model expected, checkpoint did not provide).
• Unexpected keys (checkpoint had, model did not need).
• Shape mismatches.
• Skipped keys (e.g., dropped because of a WeightConverter rename).

The report is rendered to the logger; pass transformers.logging.set_verbosity_debug() to see every detail.

Integration points

• Auto classes — auto factories layer on top of from_pretrained.
• Modeling — PreTrainedModel defines the method.
• Quantization — HfQuantizer plugs into the lifecycle.
• Tensor parallelism — tp_plan is honoured during loading.
• Modular models — modular shards still produce the same from_pretrained surface.

Entry points for modification

• Conversion ops → add a class to src/transformers/core_model_loading.py (Concatenate, Transpose, WeightRenaming, MergeModulelist, …).
• Per-architecture conversion mapping → src/transformers/conversion_mapping.py.
• Quantization lifecycle → src/transformers/quantizers/base.py.
• Hub helpers → src/transformers/utils/hub.py.