Features

This section covers cross-cutting capabilities that span multiple systems. Where a feature has a dominant home (e.g., the scheduler owns prefix caching), it gets a section there too — these pages emphasize the user-visible knobs and the integration story.

• Quantization — FP8, MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO
• LoRA serving — multi-LoRA, MoE LoRA, hot-swap, filesystem and HF Hub resolvers
• Prefix caching — block hashing, sharing, eviction, salting
• Speculative decoding — n-gram, suffix, EAGLE, MTP, draft models
• Structured outputs — JSON / regex / grammar / choice constrained sampling
• Disaggregated prefill / KV transfer — splitting prefill and decode
• Tool calling and reasoning — tool parsers, reasoning parsers
• Pooling tasks — embeddings, scoring, classification, reranking