Testing

Layout

tests/ is organized by subsystem. Notable directories:

Directory	Covers
tests/v1/	V1 engine, scheduler, KV cache, async LLM, structured output
tests/v1/engine/	EngineCore, AsyncLLM, LLMEngine, output processor
tests/v1/core/	Scheduler, KV cache manager, encoder cache, sample/spec_decode
tests/v1/worker/	GPU/CPU worker and model runner
tests/v1/attention/	Attention backend correctness and MLA paths
tests/models/	Model registry, per-architecture smoke tests, multi-modal models
tests/distributed/	TP/PP/DP across multiple processes (run only in distributed shards)
tests/entrypoints/	Offline LLM, OpenAI server, Anthropic, sagemaker, MCP
tests/lora/	LoRA loading, hot-swap, MoE LoRA
tests/quantization/	All quant formats: FP8, MXFP4, NVFP4, AWQ, GPTQ, GGUF, modelopt
tests/spec_decode/	Speculative decoding (n-gram, EAGLE, MTP, draft model)
tests/kernels/	Direct kernel correctness/perf for cache, attention, layernorm, etc.
tests/multimodal/	Vision/audio/video processing pipelines
tests/compile/	torch.compile + CUDA graph capture and replay
tests/tool_use/, tests/tokenizers_/, tests/structured_outputs/	Frontend feature areas

The fixtures and helpers used across the suite live in tests/conftest.py and per-subdir conftest.py files.

Markers

Markers are declared in pyproject.toml under [tool.pytest.ini_options]:

markers = [
    "slow_test",
    "skip_global_cleanup",
    "core_model",
    "hybrid_model",
    "cpu_model",
    "cpu_test",
    "split",
    "distributed",
    "optional",
]

Common combinations:

# Skip slow tests
pytest tests/ -m "not slow_test"

# Run only the core-model subset (PR gate)
pytest tests/models/ -m "core_model"

# CPU-only run (no GPU)
pytest tests/ -m "cpu_test"

# Include optional tests
pytest tests/ --optional

Running locally vs. in CI

Local laptops cannot reasonably run the full suite. Useful shortcuts:

# Smoke test the engine
.venv/bin/python -m pytest tests/v1/engine/test_async_llm.py -v

# Verify a model architecture
.venv/bin/python -m pytest tests/models/language/generation/test_qwen.py -v

# Benchmark sanity (no API server)
.venv/bin/python tests/v1/engine/test_engine_core.py

The full matrix runs in Buildkite (.buildkite/). The pipeline is split into many shards; sharding metadata is generated by helpers in .buildkite/scripts/ and uses the split pytest marker.

Distributed tests

Tests marked distributed need 2+ GPUs (and often NCCL). They are skipped on the default runner and only execute in dedicated multi-GPU shards. Tests under tests/distributed/ and several files under tests/v1/ and tests/lora/ set this marker.

If you must run distributed tests locally:

.venv/bin/python -m pytest tests/distributed/test_tensor_parallel.py -v -s -m distributed

You will need --tensor-parallel-size 2 (or higher) supported hardware.

Models registry

pre-commit run model-registry validates that every architecture in vllm/model_executor/models/registry.py has a corresponding implementation file and entry. Adding a new model means:

1. Add the implementation under vllm/model_executor/models/.
2. Register it in vllm/model_executor/models/registry.py.
3. Add a smoke test under tests/models/.
4. Run the registry hook before pushing.

Quick reference

# Make pytest verbose and stop at first failure
pytest -xvs tests/path/to/test_file.py

# Show captured stdout for a passing test
pytest tests/v1/engine/test_async_llm.py -v --capture=no

# Filter by keyword
pytest tests/ -k "prefix_caching" -v

For tracing and debugging, see Debugging.