Debugging

Logs

vLLM uses a custom logger initialized via vllm.logger.init_logger(name) (vllm/logger.py). Two environment variables matter most:

Variable	Effect
VLLM_LOGGING_LEVEL	Sets the global log level (DEBUG, INFO, WARNING, ERROR)
VLLM_LOGGING_CONFIG_PATH	Path to a JSON file with a Python logging config dict

The full set of envs and their semantics live in vllm/envs.py (~2,500 lines, all environment overrides documented inline).

Engine input dumps

When the EngineCore catches an exception while processing a request, it can dump the offending input to disk. This is gated by VLLM_LOG_DUMP_PATH (or the equivalent ObservabilityConfig knob). The implementation is vllm/logging_utils/dump_input.py's dump_engine_exception. The dump captures the EngineCoreRequest and the surrounding scheduler state — useful when reproducing flaky failures from production.

Common error categories

Symptom	Likely area
EngineDeadError / EngineGenerateError	EngineCore process crashed; check stderr for the underlying CUDA / OOM error
OutOfMemoryError during model loading	Set VLLM_USE_PRECOMPILED=1, lower --gpu-memory-utilization, or shard with TP
Hang during startup	Often forkserver / multiprocessing setup. VLLM_WORKER_MULTIPROC_METHOD=spawn
NCCL timeout in distributed runs	Increase NCCL_TIMEOUT; check that all ranks reach the same code path
AssertionError: kv_cache_groups mismatch	KV cache config doesn't match attention layer specs; usually an attention backend mismatch
RuntimeError: ... CUDAGraph captured ...	The model has a non-graph-safe op; flip --compilation-config '{"cuda_graph_mode": "PIECEWISE"}' or NONE

Tracing a request end-to-end

1. Frontend: log the parsed request in vllm/entrypoints/openai/chat_completion/serving.py (or the equivalent serving.py for the endpoint you care about).
2. InputProcessor: vllm/v1/engine/input_processor.py — where prompts are tokenized and EngineCoreRequest is built.
3. AsyncLLM: vllm/v1/engine/async_llm.py — _add_request enqueues to the EngineCore client.
4. EngineCore loop: vllm/v1/engine/core.py — add_request, then step.
5. Scheduler: vllm/v1/core/sched/scheduler.py::schedule — picks running set; update_from_output applies results.
6. Worker: vllm/v1/worker/gpu_worker.py and gpu_model_runner.py::execute_model.
7. OutputProcessor: vllm/v1/engine/output_processor.py — builds RequestOutput chunks and sends them back.

A breakpoint at any of these stages is a good way to learn the flow. Set VLLM_LOGGING_LEVEL=DEBUG and most steps will print already.

OpenTelemetry

If --otlp-traces-endpoint is configured (ObservabilityConfig.otlp_traces_endpoint), spans are emitted from key code paths via vllm/tracing.py. The decorator @instrument(span_name=...) is used liberally across the engine.

Profiling

Two main paths:

• Torch profiler. Triggered via LLM.start_profile() / LLM.stop_profile() (which fan out via Executor.profile). Output is a Chrome trace.
• NVTX hooks. vllm/utils/nvtx_pytorch_hooks.py decorates many internals with NVTX ranges so Nsight Systems / Nsight Compute traces stay readable.

When all else fails

• vllm.collect_env (also vllm collect-env) prints a full environment report — paste it into bug reports.
• The #users and #dev channels on Slack are responsive.
• Reproductions on tiny models (Qwen/Qwen3-0.6B, facebook/opt-125m) are strongly preferred when filing issues.