OpenAI-compatible server

Active contributors: Harry Mellor, Cyrus Leung, Nick Hill, Roger Wang.

Purpose

vllm serve launches a FastAPI app that speaks the OpenAI HTTP API (plus a few extensions). Clients written for OpenAI, vendor-managed wrappers, and frameworks like LiteLLM, LangChain, and LlamaIndex talk to it without modification.

Where it lives

vllm/entrypoints/openai/
├── api_server.py         # FastAPI app and request routing (719 lines)
├── cli_args.py           # CLI argument schema (~17 KB)
├── server_utils.py       # Lifespan, error handlers, log middleware (~17 KB)
├── launcher.py           # Uvicorn launcher
├── models/               # /v1/models implementation
├── chat_completion/      # /v1/chat/completions
├── completion/           # /v1/completions
├── generate/             # Generic /generate
├── generative_scoring/   # /v1/score, /v1/rerank
├── responses/            # /v1/responses
├── realtime/             # WebSocket /v1/realtime
├── speech_to_text/       # /v1/audio/transcriptions, /translations
├── parser/               # Tool-call parsing helpers
├── engine/               # EngineProtocol implementation
├── run_batch.py          # /v1/batches (file-based async batches)
├── orca_metrics.py       # ORCA-format per-request metrics headers
├── fingerprint.py        # System fingerprint generation
└── utils.py

The Anthropic Messages, gRPC, MCP, SageMaker, and pooling endpoints share helpers from this tree but live in vllm/entrypoints/{anthropic,mcp,sagemaker,pooling}/ and vllm/entrypoints/grpc_server.py.

Endpoints

Method	Path	Behavior
POST	/v1/chat/completions	OpenAI chat (with tools, JSON mode, streaming, vision)
POST	/v1/completions	OpenAI legacy completions
POST	/v1/embeddings	Embedding inference (pooling)
POST	/v1/score	Pairwise scoring
POST	/v1/rerank	Reranking
POST	/v1/classify	Classification (pooling)
POST	/v1/audio/transcriptions	Whisper-style ASR
POST	/v1/audio/translations	Whisper translation
POST	/v1/responses	OpenAI Responses API
WS	/v1/realtime	OpenAI realtime API (streaming speech-to-text/-from-text)
POST	/tokenize / /detokenize	Tokenizer access
POST	/v1/batches, GET /v1/batches/{id}	File-based async batch jobs
GET	/v1/models	List served models
GET	/health	Healthcheck
GET	/version	Version + system fingerprint
GET	/metrics	Prometheus exposition
GET/POST	LoRA add/remove endpoints	Hot-load adapters via REST

Request flow

sequenceDiagram
    participant Cli as Client
    participant Mid as ASGI middleware<br/>(CORS, logging, scaling, request-id)
    participant Route as FastAPI route<br/>(chat_completion/serving.py)
    participant Tool as ToolParser
    participant Reason as ReasoningParser
    participant Async as AsyncLLM
    participant OP as OutputProcessor
    Cli->>Mid: HTTP POST /v1/chat/completions
    Mid->>Route: parsed body
    Route->>Route: chat template (chat_utils.py) + tools setup
    Route->>Async: generate(prompt, sampling_params, tools, structured_outputs)
    loop streaming
        Async->>OP: EngineCoreOutput
        OP->>Route: RequestOutput chunk
        Route->>Tool: tool delta extraction
        Route->>Reason: reasoning delta extraction
        Route-->>Cli: SSE chunk
    end
    Route-->>Cli: terminating chunk + usage

Argument surface

vllm/entrypoints/openai/cli_args.py::make_arg_parser builds the CLI argument schema. It overlays:

• HTTP-side options (host, port, ssl, CORS, served-model-name, response-role)
• Engine options (auto-generated from AsyncEngineArgs)
• Auth (API key, bearer)
• LoRA preload (--lora-modules)
• Tool / reasoning parser names
• Tracing endpoint
• Multi-server replication (--api-server-count, --worker-multiproc-method)

validate_parsed_serve_args ensures the combinations make sense (e.g., --headless precludes --api-server-count > 0).

Lifespan

server_utils.py::lifespan runs:

1. setup_server — argument validation, socket binding, prometheus setup.
2. Spawn engine via build_async_engine_client.
3. Yield control to FastAPI (serve requests).
4. On shutdown: graceful drain, shutdown_prometheus, engine teardown.

The Uvicorn launcher (vllm/entrypoints/launcher.py::serve_http) sets up signal handlers and the worker count.

Multi-process API server

--api-server-count N runs N API server processes in parallel against a single EngineCore. Implementation:

• APIServerProcessManager (vllm/v1/utils.py) spawns and supervises the servers.
• Each process writes a unique client_index; outputs route back via that index.
• Prometheus uses multiprocess mode to merge metrics across processes.

This is the recommended deployment when one CPU process bottlenecks request parsing / templating before it hits the engine.

Authentication

The server supports:

• --api-key <key> — single-key bearer auth
• --api-key-file <path> — multiple keys from a file
• TLS via --ssl-keyfile / --ssl-certfile / --ssl-ca-certs
• IPv6 via is_valid_ipv6_address detection

ORCA load reporting

orca_metrics.py produces ORCA-format response headers (CPU utilization, memory, requests-in-flight) for use behind Envoy / Istio for load-aware routing.

Key source files

File	Purpose
vllm/entrypoints/openai/api_server.py	FastAPI app + route registration
vllm/entrypoints/openai/cli_args.py	CLI schema
vllm/entrypoints/openai/server_utils.py	Lifespan, error handlers
vllm/entrypoints/openai/chat_completion/serving.py	Chat completion handler
vllm/entrypoints/openai/completion/serving.py	Completion handler
vllm/entrypoints/openai/responses/	Responses API handler
vllm/entrypoints/openai/realtime/	Realtime WS handler
vllm/entrypoints/openai/run_batch.py	/v1/batches
vllm/entrypoints/launcher.py	Uvicorn launcher
vllm/entrypoints/openai/orca_metrics.py	ORCA headers

Entry points for modification

• New endpoint: write a handler under vllm/entrypoints/openai/<topic>/, register a router in api_server.py, add tests in tests/entrypoints/openai/.
• New chat-template logic: extend vllm/entrypoints/chat_utils.py.
• New auth scheme: middleware in server_utils.py.
• New header / metric: orca_metrics.py is a good template.