Configuration

Purpose

Every architecture in transformers describes its hyperparameters with a PretrainedConfig subclass. The base class handles JSON serialization, Hub upload/download, attribute defaults, and the migration of deprecated keys. Configurations are the unit of comparison the auto factories use to choose a concrete model class.

Key abstractions

Class	File	Role
PretrainedConfig (alias PreTrainedConfig)	src/transformers/configuration_utils.py	Base class for all configs
<Arch>Config	src/transformers/models/<arch>/configuration_<arch>.py	One per architecture
AutoConfig	src/transformers/models/auto/configuration_auto.py	Late-binding factory
CONFIG_MAPPING	src/transformers/models/auto/configuration_auto.py	model_type → config class registry

Anatomy of a config class

class LlamaConfig(PretrainedConfig):
    model_type = "llama"
    keys_to_ignore_at_inference = ["past_key_values"]

    def __init__(self, vocab_size=..., hidden_size=..., ...):
        super().__init__(...)
        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        ...

model_type is the discriminator the auto-factory uses; CONFIG_MAPPING[config.model_type] returns the class. keys_to_ignore_at_inference lists output keys the auto-factory should not surface in inference outputs.

JSON I/O

PretrainedConfig.to_dict() serializes the dataclass-style attributes; from_dict() reverses it. The on-disk format is config.json (and generation_config.json for generation defaults). The Hub round-trip:

cfg = LlamaConfig.from_pretrained("meta-llama/Llama-2-7b-hf")
cfg.save_pretrained("./my-llama")
cfg.push_to_hub("my-org/my-llama")

The Hub helpers come from src/transformers/utils/hub.py.

Deprecated kwargs and config migrations

Old checkpoints often contain attribute names that have since been renamed. PretrainedConfig._handle_deprecated_kwargs (and per-model overrides) maps them to the new names with a warning. This keeps the Backwards Compatibility tenet honest.

When you need to introduce a renamed attribute, follow the existing pattern:

def __init__(self, hidden_dim=None, hidden_size=None, **kwargs):
    if hidden_dim is not None:
        warnings.warn("hidden_dim is deprecated; use hidden_size", FutureWarning)
        hidden_size = hidden_dim
    ...

Sub-configs

Many configs nest other configs (e.g., a vision encoder + text decoder for multimodal models). The base class supports nested PretrainedConfig instances and serializes them transparently.

Generation config

generation_config.json is loaded into a separate GenerationConfig object (src/transformers/generation/configuration_utils.py, 103K LOC) when the model is loaded. This avoids polluting model configs with decoding knobs and makes generation defaults shareable across multiple checkpoints.

Integration points

• The Hub layer (src/transformers/utils/hub.py) reads/writes config.json and generation_config.json.
• The auto factories use CONFIG_MAPPING[config.model_type] to select model classes.
• PreTrainedModel.from_pretrained consumes a PretrainedConfig (or builds one from disk) before instantiating the module.
• Trainer accesses model.config for attributes like pad_token_id and tie_word_embeddings.

Entry points for modification

To add a new architecture, add a <Arch>Config in src/transformers/models/<arch>/configuration_<arch>.py, register it in auto_mappings.py, and run make fix-repo. To deprecate a config attribute, follow the FutureWarning pattern shown above and add the rename mapping to _handle_deprecated_kwargs in the per-model class.