Serving

transformers serve exposes a model over HTTP with an OpenAI-compatible API. It is built on FastAPI, integrates with continuous batching for throughput, and supports text generation, vision-language input, ASR, and tool calling.

At a glance

pip install "transformers[torch,serving]"
transformers serve --port 8000 --continuous-batching

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

Where the code lives

File / dir	Purpose
src/transformers/cli/serve.py	Typer entry point and CLI argument definitions
src/transformers/cli/serving/	FastAPI app, route handlers, middleware
src/transformers/cli/serving/utils.py	Helpers (set_torch_seed, etc.)
src/transformers/generation/continuous_batching/	Scheduler used when --continuous-batching is on
src/transformers/generation/streamers.py	AsyncTextStreamer for SSE responses

The implementation requires the serving extras (fastapi, uvicorn, httpx, etc.). Whether the dependency is available is checked by transformers.utils.import_utils.is_serve_available.

Endpoints

Method	Path	Purpose
POST	/v1/chat/completions	OpenAI Chat Completions (streaming via SSE supported).
POST	/v1/completions	Legacy completions.
POST	/v1/responses	OpenAI Responses API.
POST	/v1/audio/transcriptions	Whisper-style ASR.
GET	/v1/models	List models known to the server.
GET	/health, /v1/health	Liveness probe.

Key flags

transformers serve [MODEL_REPO_ID]
  --host 0.0.0.0
  --port 8000
  --force-model <repo>
  --continuous-batching
  --cb-block-size <int>
  --cb-num-blocks <int>
  --cb-max-batch-tokens <int>
  --enable-tools
  --enable-vision
  --max-concurrency <int>
  --ssl-keyfile <path> --ssl-certfile <path>
  --api-key <secret>
  --log-level <info|debug|...>

--force-model pins the server to a single checkpoint and refuses to swap. Without it, the server lazily loads checkpoints by request model field (subject to local cache and HF_HOME).

Streaming

Set "stream": true in the request body. The server returns text/event-stream with chunked deltas matching the OpenAI shape:

data: {"id":"...","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"}}]}

data: {"id":"...","choices":[{"finish_reason":"stop"}]}

data: [DONE]

Implementation uses AsyncTextStreamer to push tokens as they are decoded.

Continuous batching mode

Pass --continuous-batching to enable the paged-KV scheduler from src/transformers/generation/continuous_batching/. The server then admits multiple concurrent requests into a single forward pass, dramatically improving throughput for variable-length workloads. See Continuous batching.

Tool calling and vision

• --enable-tools accepts tools=[...] in the request and instructs the model via the chat template. The response shape mirrors OpenAI's tool_calls array; the parser is src/transformers/utils/chat_parsing_utils.py.
• --enable-vision accepts image inputs in messages[].content[] parts (URL or base64). The server routes through the matching Processor to render images.

Authentication

--api-key <secret> enables a simple bearer-token check on every request. For production deployments, terminate TLS upstream (e.g., behind an Nginx or cloud load balancer) and pass the API key via Authorization: Bearer ....

Logging and metrics

The server uses Python logging (transformers.cli.serving). Metrics for queue depth, batch occupancy, and per-request latency can be scraped by hooking into the scheduler events; the surface is intentionally small at the moment.

Comparing to vLLM / TGI

transformers serve reuses the same per-architecture code paths as model.generate, which means new architectures supported by transformers are immediately serveable. It does not currently match vLLM/TGI on every micro-benchmark — those projects ship custom CUDA kernels and aggressive batching heuristics. For the highest throughput, those projects often consume transformers model definitions; for ergonomic / always-up-to-date single-binary serving, transformers serve is the right tool.

Integration points

• CLI — transformers serve is registered as a Typer command.
• Continuous batching — paged scheduler.
• Generation — the underlying decoding loop.
• Chat templates — applied to incoming messages.
• Pipelines — the non-continuous-batching path can fall back to the text-generation and ASR pipelines.

Entry points for modification

• New endpoint → add a route under src/transformers/cli/serving/ and register in the FastAPI app constructed by Serve.
• New CLI flag → extend the Serve class signature in src/transformers/cli/serve.py.
• New scheduler heuristic → src/transformers/generation/continuous_batching/scheduler.py.
• Tests → tests/cli/test_serving*.py.