Data

Purpose

Data handling in transformers is intentionally thin. The library prefers to delegate dataset I/O to the datasets library and focus on the preprocessing → batching contract that Trainer consumes. This subsystem hosts data collators, legacy GLUE/SQuAD processors, and the HfArgumentParser helper.

Key abstractions

Class / function	File	Role
default_data_collator, DefaultDataCollator	src/transformers/data/data_collator.py	Pad tensors to the longest sequence in the batch
DataCollatorWithPadding	same	Padding driven by a tokenizer
DataCollatorWithFlattening	same	Pack examples into a single flat sequence
DataCollatorForLanguageModeling	same	MLM masking (BERT-style)
DataCollatorForWholeWordMask	same	Whole-word MLM
DataCollatorForPermutationLanguageModeling	same	XLNet-style
DataCollatorForSeq2Seq	same	Pad inputs and labels
DataCollatorForTokenClassification	same	Pad labels with -100
DataCollatorForMultipleChoice	same	Pad multi-choice examples
DataCollatorForSOP	same	Sentence-order prediction
DataProcessor, InputExample, InputFeatures, SquadExample, SquadFeatures	src/transformers/data/processors/	Legacy GLUE/SQuAD/XNLI processors
HfArgumentParser	src/transformers/hf_argparser.py (20K LOC)	Argparse for dataclasses (used by example scripts)

Data collators

A collator is a callable that takes a list of dataset items and returns a single batched dict of tensors. The choice of collator is task-specific:

• default_data_collator — every item is already a tensor of equal length; just stack.
• DataCollatorWithPadding — pad text inputs to the longest in the batch.
• DataCollatorForLanguageModeling(mlm=True, mlm_probability=0.15) — replaces 15% of tokens with [MASK] for BERT/RoBERTa pretraining.
• DataCollatorForSeq2Seq — pad encoder inputs and decoder labels separately.
• DataCollatorForTokenClassification — pad labels with -100 so they're ignored by cross-entropy.
• DataCollatorWithFlattening — concatenate examples and emit position_ids for sample packing (used in efficient SFT).

from transformers import DataCollatorForLanguageModeling, AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
collator = DataCollatorForLanguageModeling(tokenizer=tok, mlm_probability=0.15)
trainer = Trainer(model=..., args=..., train_dataset=..., data_collator=collator)

Legacy GLUE / SQuAD processors

Pre-datasets examples relied on hand-written processors in src/transformers/data/processors/glue.py, squad.py, xnli.py. They are kept for backward compatibility but new code should use datasets directly. The associated metrics (glue_compute_metrics, etc.) live in src/transformers/data/metrics/.

HfArgumentParser

src/transformers/hf_argparser.py (20K LOC) extends argparse.ArgumentParser to read fields from a dataclass and produce a fully-typed args object. Every examples/pytorch/<task>/run_*.py script uses this pattern:

parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
model_args, data_args, training_args = parser.parse_args_into_dataclasses()

It also supports JSON / YAML config files via parser.parse_json_file() / parser.parse_yaml_file().

Sample packing

DataCollatorWithFlattening is the sample-packing entry point. Combined with attention masks built by src/transformers/masking_utils.py, it lets Trainer train on packed sequences without losing per-sample boundaries (each "row" still attends only to itself thanks to block-diagonal masks).

Padding strategies

Tokenizers expose padding="longest" | "max_length" | False and truncation=True | "longest_first" | "only_first" | "only_second". Collators respect these by re-padding within the batch. For static-shape compile/serving, prefer padding="max_length" so every batch has the same shape.

Integration points

• Trainer — the data_collator argument.
• Pipelines — pipelines do their own batching and do not use data_collator.
• examples/pytorch/<task>/ scripts use HfArgumentParser to wire dataclasses into Trainer.

Entry points for modification

• Add a new collator → put a class in src/transformers/data/data_collator.py and re-export it from src/transformers/__init__.py.
• Sample-packing improvements → edit DataCollatorWithFlattening and the matching mask builder in src/transformers/masking_utils.py.
• Reference scripts → examples/pytorch/<task>/run_<task>.py and friends.