vLLM, a high-throughput inference engine from UC Berkeley, now maintained by 2000+ contributors, uses PagedAttention to efficiently serve 200+ model architectures across diverse hardware.

What it is

vLLM is an open-source library for LLM inference and serving, originally developed at UC Berkeley’s Sky Computing Lab and now maintained by a community of over 2000 contributors across academic institutions and companies. The project has reached 95,645 stars on GitHub with 24,021 forks.

The core innovation: PagedAttention

vLLM’s central optimization is PagedAttention, which manages attention key and value memory like virtual memory in operating systems. Instead of allocating contiguous blocks, it pages in memory as needed, reducing fragmentation and enabling higher throughput. The engine combines this with continuous batching of incoming requests, chunked prefill, and prefix caching to maximize utilization.

Additional performance features include fast and flexible model execution with piecewise and full CUDA/HIP graphs, speculative decoding (n-gram, suffix, EAGLE, DFlash variants), and optimized attention kernels including FlashAttention, FlashInfer, and FlashMLA.

Breadth of support

vLLM seamlessly supports 200+ model architectures on Hugging Face, including decoder-only LLMs (Llama, Qwen, Gemma), mixture-of-expert models (Mixtral, DeepSeek-V3, Qwen-MoE), hybrid attention and state-space models (Mamba, Qwen3.5), multi-modal models (LLaVA, Qwen-VL, Pixtral), and embedding and retrieval models.

Quantization options span FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF, and more. Hardware support includes NVIDIA and AMD GPUs, Intel GPUs, Google TPUs, Intel Gaudi, Apple Silicon, and x86/ARM/PowerPC CPUs.

The API surface includes OpenAI-compatible endpoints, Anthropic Messages API, gRPC support, multi-LoRA for dense and MoE layers, structured output generation via xgrammar or guidance, and tool calling and reasoning parsers.

Getting started

Install with uv pip install vllm. The README recommends running vllm download-kernels afterward to precompile kernels for faster startup. Full documentation, model lists, and installation guides are available at vllm.ai.

Go deeper

Get the latest AI updates in the Tech AI Magazine app
iPhone · Android