Front-ends and entry points

Active contributors: Cyrus Leung, Harry Mellor, Nick Hill, Roger Wang.

Purpose

Everything that converts an external request (Python call, HTTP request, gRPC call, MCP message) into an EngineCoreRequest lives under vllm/entrypoints/. There are five front-ends that all share the same engine core.

Directory layout

vllm/entrypoints/
├── llm.py                    # Offline LLM API (1,935 lines)
├── api_server.py             # Smaller convenience server
├── chat_utils.py             # Chat template handling (62 KB)
├── launcher.py               # Uvicorn launcher used by openai/api_server
├── grpc_server.py            # gRPC entry point
├── logger.py                 # RequestLogger
├── ssl.py                    # TLS config helpers
├── utils.py                  # cli_env_setup, log_non_default_args, etc.
├── cli/                      # The `vllm` CLI
│   ├── main.py               # Dispatcher
│   ├── serve.py              # `vllm serve`
│   ├── openai.py             # `vllm complete`, `vllm chat`
│   ├── benchmark/            # `vllm bench`
│   ├── collect_env.py        # `vllm collect-env`
│   ├── launch.py             # `vllm launch`
│   ├── run_batch.py          # `vllm run-batch`
│   └── types.py
├── openai/                   # OpenAI-compatible REST server
│   ├── api_server.py         # FastAPI app (719 lines)
│   ├── cli_args.py           # `vllm serve` argument schema
│   ├── server_utils.py       # Lifespan, error handlers, log middleware
│   ├── chat_completion/, completion/, generate/, generative_scoring/
│   ├── responses/, realtime/, speech_to_text/, models/, parser/
│   ├── engine/               # EngineProtocol implementation for OpenAI server
│   └── run_batch.py          # /v1/batches implementation
├── anthropic/                # Anthropic Messages API
├── pooling/                  # Pooling (embed/score/classify/rerank) helpers
├── sagemaker/                # AWS SageMaker custom inference endpoint
├── mcp/                      # Model Context Protocol server
├── serve/                    # Auxiliary serving middleware
└── realtime/                 # WebSocket realtime endpoints

Key abstractions

Abstraction	File	Role
LLM	vllm/entrypoints/llm.py	Offline batch interface; returns lists of RequestOutput
OpenAIServingChat, OpenAIServingCompletion, etc.	vllm/entrypoints/openai/.../serving.py	Request-handler classes per OpenAI endpoint
EngineProtocol	vllm/entrypoints/openai/engine/protocol.py	Adapter the OpenAI handlers talk to (engine-agnostic)
RequestLogger	vllm/entrypoints/logger.py	Optional HTTP request logger
serve_http	vllm/entrypoints/launcher.py	Wraps the FastAPI app in a uvicorn server with proper signals
CLISubcommand	vllm/entrypoints/cli/types.py	Base class for vllm <subcommand> modules
cmd_init()	(per CLI module)	Each CLI subcommand exposes cmd_init() returning a list of CLISubcommand instances

How vllm serve starts up

sequenceDiagram
    participant User
    participant CLI as vllm CLI<br/>(entrypoints/cli/main.py)
    participant Serve as ServeSubcommand<br/>(entrypoints/cli/serve.py)
    participant API as OpenAI server<br/>(entrypoints/openai/api_server.py)
    participant Async as AsyncLLM<br/>(v1/engine/async_llm.py)
    participant Core as EngineCore<br/>(v1/engine/core.py)
    participant W as Workers<br/>(v1/worker/gpu_worker.py)

    User->>CLI: vllm serve <model>
    CLI->>Serve: dispatch_function(args)
    Serve->>API: setup_server, run_server
    API->>Async: build_async_engine_client_from_engine_args
    Async->>Core: spawn EngineCore proc(es) (CoreEngineProcManager)
    Core->>W: launch workers (MultiprocExecutor / Ray)
    W-->>Core: KV cache spec, available memory
    Core-->>Async: EngineCoreReadyResponse
    API->>API: start FastAPI app on uvicorn
    User->>API: POST /v1/chat/completions
    API->>Async: generate(...)
    Async->>Core: EngineCoreRequest (ZMQ)
    Core->>W: SchedulerOutput per step
    W-->>Core: ModelRunnerOutput
    Core-->>Async: EngineCoreOutputs (ZMQ)
    Async-->>API: stream RequestOutput
    API-->>User: SSE chunks

The two-step setup_server / run_server split comes from vllm/entrypoints/openai/api_server.py. setup_server does early validation and listens on the socket; run_server builds the engine client, mounts routes, and enters the uvicorn loop.

OpenAI endpoint coverage

vllm/entrypoints/openai/api_server.py mounts (subject to model capabilities):

• POST /v1/chat/completions
• POST /v1/completions
• POST /v1/embeddings
• POST /v1/score, POST /v1/rerank
• POST /v1/classify
• POST /v1/audio/transcriptions, POST /v1/audio/translations
• POST /v1/responses (the OpenAI Responses API)
• WebSocket /v1/realtime (OpenAI realtime API for streaming speech)
• POST /v1/batches, GET /v1/batches/{id}
• POST /tokenize, POST /detokenize
• GET /v1/models, GET /health, GET /version, GET /metrics

Each endpoint has a corresponding OpenAIServing* class in the matching subdirectory (chat_completion/, completion/, responses/, etc.) that converts the OpenAI request into an engine call.

The Anthropic Messages API (vllm/entrypoints/anthropic/), gRPC (vllm/entrypoints/grpc_server.py), MCP (vllm/entrypoints/mcp/), and SageMaker (vllm/entrypoints/sagemaker/) front-ends provide alternative shells over the same engine.

API server replication

For high-fanout deployments, --api-server-count N runs N FastAPI processes that all talk to the same EngineCore. The orchestrator is APIServerProcessManager (vllm/v1/utils.py). Each replica gets its own client_index; outputs are routed back to the originating replica via the client_index field on EngineCoreOutput.

Integration points

• AsyncLLM (vllm/v1/engine/async_llm.py) is the contract every front-end uses for streaming generation. Anything that needs to call into the engine constructs an AsyncLLM instance.
• InputProcessor (vllm/v1/engine/input_processor.py) tokenizes prompts, applies chat templates (via chat_utils.py), processes multimodal inputs (via vllm/multimodal/), and emits an EngineCoreRequest.
• OutputProcessor (vllm/v1/engine/output_processor.py) detokenizes engine outputs, applies sampling-style finalization (stop strings, tool parsers, reasoning parsers), and produces RequestOutput chunks.

Entry points for modification

• Add an HTTP endpoint: subclass an OpenAIServing* (or write a new one), add a router in vllm/entrypoints/openai/api_server.py, and add tests under tests/entrypoints/openai/.
• Add a CLI subcommand: write a module in vllm/entrypoints/cli/ that exports cmd_init() returning [CLISubcommand], and register it in vllm/entrypoints/cli/main.py::CMD_MODULES.
• Add a tool parser: drop a module under vllm/tool_parsers/ that registers itself with ToolParserManager.
• Add a reasoning parser: same pattern under vllm/reasoning/ with ReasoningParserManager.

See Engine core for what happens after the front-end hands off.