Testing

The test surface mirrors the source tree and is one of the largest in any open-source ML project: ~412K lines of Python under tests/.

Layout

tests/
├── models/
│   └── <name>/
│       ├── test_modeling_<name>.py
│       ├── test_tokenization_<name>.py
│       ├── test_processor_<name>.py
│       └── test_image_processing_<name>.py
├── pipelines/
│   └── test_pipelines_<task>.py
├── generation/
├── quantization/
│   └── <backend>/
├── tensor_parallel/
├── trainer/
├── repo_utils/
├── sagemaker/
├── kernels/
├── peft_integration/
├── optimization/
├── utils/
├── fixtures/                 # shared test inputs
├── test_modeling_common.py   # 291K LOC — shared mixin
├── test_pipeline_mixin.py    # 38K LOC
├── test_processing_common.py # 95K LOC
├── test_tokenization_common.py # 133K LOC
├── causal_lm_tester.py       # base tester for autoregressive LMs
├── vlm_tester.py             # base tester for vision-language models
└── conftest.py               # global fixtures

Mixins do most of the work

Per-model test files are typically thin wrappers around ModelTesterMixin from tests/test_modeling_common.py. Each model's test file declares the small set of all_model_classes, all_generative_model_classes, an init dict, and the mixin runs ~80 standardized checks (forward, backward, config save/load, gradient checkpointing, equivalence of attention backends, scripting, training, etc.).

Common mixins:

Mixin	File	Purpose
ModelTesterMixin	tests/test_modeling_common.py	Modeling sanity checks
GenerationTesterMixin	tests/test_modeling_common.py	generate checks
TrainingTesterMixin	tests/test_training_mixin.py	Trainer integration
TensorParallelTesterMixin	tests/test_tensor_parallel_mixin.py	TP correctness
ProcessorTesterMixin	tests/test_processing_common.py	Multimodal processors
TokenizerTesterMixin	tests/test_tokenization_common.py	Tokenizer round-trips
ImageProcessorTesterMixin	tests/test_image_processing_common.py	Image preprocessor
PipelineTesterMixin	tests/test_pipeline_mixin.py	Per-task pipelines
CausalLMTester	tests/causal_lm_tester.py	Standardized causal-LM checks
VLMTester	tests/vlm_tester.py	Standardized VLM checks

Running tests

Smoke test for a single model

pytest tests/models/llama/
pytest tests/models/llama/test_modeling_llama.py -k "generate"

Slow tests

Many tests require GPU and large model downloads. They are skipped by default and gated by RUN_SLOW=1:

RUN_SLOW=1 pytest tests/models/llama/

Test selectors and markers

Markers declared in pyproject.toml [tool.pytest.ini_options]:

Marker	Selects
flash_attn_test, flash_attn_3_test, flash_attn_4_test, all_flash_attn_test	FlashAttention variants
bitsandbytes	bitsandbytes integration
generate	GenerationTesterMixin
is_training_test	TrainingTesterMixin
is_tensor_parallel_test	TensorParallelTesterMixin

pytest -m "generate and not slow"

Hardware/dependency requires

Defined in src/transformers/testing_utils.py (159K LOC):

• @require_torch_gpu, @require_torch_multi_gpu, @require_torch_accelerator.
• @require_flash_attn, @require_flash_attn_3, @require_flash_attn_4.
• @require_bitsandbytes, @require_auto_gptq, @require_torchao, etc. — one per quantizer.
• @require_peft, @require_accelerate, @require_deepspeed.
• @require_vision, @require_torchaudio, @require_av.

These decorators short-circuit tests when the dependency or hardware is missing.

Tiny model fixtures

Most fast tests run on shrunk-down fixture models (a few thousand parameters) created by utils/create_dummy_models.py (84K LOC). The fixtures live on the Hub under the hf-internal-testing org and are downloaded on demand. The script utils/update_tiny_models.py regenerates them.

Doctest

Markdown docs and select Python files include doctests. Configuration:

• pyproject.toml enables --doctest-glob='**/*.md' with NUMBER NORMALIZE_WHITESPACE ELLIPSIS flags.
• The opt-in list of files to doctest is utils/not_doctested.txt (inverted: files in the list are excluded). Maintained by utils/check_doctest_list.py.
• make doc_test is roughly: pytest --doctest-modules <files>.

CI: .github/workflows/doctests.yml runs doctests on a schedule.

Test fetcher

utils/tests_fetcher.py (52K LOC) computes which test files are likely to be affected by a branch's changes. Used both locally and by CI to skip irrelevant tests.

python utils/tests_fetcher.py

It writes a test_list.txt and a examples_test_list.txt consumed by CI.

CI matrices

.github/workflows/ and .circleci/config.yml define many matrices:

• model_jobs.yml, model_jobs_intel_gaudi.yml — per-model GPU tests.
• self-scheduled-amd-mi250-caller.yml, self-scheduled-amd-mi325-caller.yml, self-scheduled-amd-mi355-caller.yml — AMD GPU runs.
• self-scheduled-flash-attn-caller.yml — FlashAttention regressions.
• self-scheduled-intel-gaudi.yml, self-scheduled-intel-gaudi3-caller.yml — Intel Gaudi.
• benchmark.yml, benchmark_v2.yml — perf tracking.
• extras-smoke-test.yml — install matrix for pip install transformers[<extra>].

Reading test failures

The notification service (utils/notification_service.py, 67K LOC) collates CI failures into Slack reports. For a local PR, the GitHub Checks UI is the entry point; click into a failing job and follow the pytest log link.