# vLLM

> Fast, memory-efficient inference and serving for large language models

- Category: Models & Inference / Serving & Deployment
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/vllm-project/vllm
- Website: https://docs.vllm.ai
- Canonical page: https://ai-tldr.dev/tools/vllm/
- Updated: 2026-11-01

## In short

vLLM is a library for running and serving large language models. It loads models from Hugging Face and exposes them either as a Python API for batch generation or as an OpenAI-compatible HTTP server. It sits in the Serving & Deployment category of the Models & Inference tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

vLLM is a library for running and serving large language models. It loads models from Hugging Face and exposes them either as a Python API for batch generation or as an OpenAI-compatible HTTP server. It was originally developed in the Sky Computing Lab at UC Berkeley and is now maintained by a large open-source community.

It is built for teams that need to serve many requests at the same time without running out of GPU memory. Its PagedAttention technique manages the attention key/value cache more efficiently, and continuous batching keeps the GPU busy by adding and removing requests on the fly. It supports 200+ model architectures, including decoder-only LLMs, mixture-of-expert models, and multimodal models.

Within the inference and serving category, vLLM sits at the high-throughput serving end. You point it at a model, and it handles batching, memory management, streaming, and an API surface so you can focus on your application instead of the serving plumbing.

## What it does

- PagedAttention manages attention key/value memory efficiently, reducing waste and fitting more concurrent requests in GPU memory
- Continuous batching, chunked prefill, and prefix caching keep throughput high under load
- OpenAI-compatible API server, plus Anthropic Messages API and gRPC support
- Wide quantization support including FP8, INT8, INT4, GPTQ/AWQ, and GGUF
- Tensor, pipeline, data, expert, and context parallelism for distributed inference
- Runs on NVIDIA and AMD GPUs, x86/ARM/PowerPC CPUs, and other hardware via plugins

## Getting started

Install vLLM, then either generate text from Python or start an OpenAI-compatible server.

1. Install vLLM

   Install with uv (recommended) or pip. A GPU with a matching CUDA setup is the common target.

```bash
uv pip install vllm
```

2. Run offline batched inference

   Load a model and generate text for a list of prompts directly in Python.

```python
from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The capital of France is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="facebook/opt-125m")
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.prompt, output.outputs[0].text)
```

3. Start an OpenAI-compatible server

   Serve a model over HTTP at http://localhost:8000 with chat and completion endpoints that match the OpenAI API.

```bash
vllm serve Qwen/Qwen2.5-1.5B-Instruct
```

## When to use it

- Serving a chat or completion model to many users at once behind an OpenAI-compatible API
- Running high-volume offline batch generation over large prompt sets
- Hosting open-weight Hugging Face models on your own GPUs instead of a paid API
- Scaling a single large model across multiple GPUs with tensor or pipeline parallelism

## Version history

- 2026-11-01 v0.33.0: Block-diffusion drafters (DFlash, DSpark) stop reserving draft slots, an opt-in lossy FLy verifier and a ROCm `--monokernel` decode path speed up generation, and a `/v1/decisions` endpoint lands. 927 commits from 389 contributors; release builds are now CUDA 13.0 only, and Medusa and the MLP speculator are removed.
- 2026-09-22 v0.30.0: Fast Start adds a per-GPU weight-cache daemon, so a restarting engine maps post-quantized weights over CUDA IPC via `--load-format ipc_cache` instead of reloading from disk. 762 commits from 315 contributors also bring Gumbel-max watermarking, the HiSparse host-memory KV tier, and DeepSeek-V4.1-Flash and GLM-5.3-Flash support. Scale-out endpoints now need `--enable-scale-out`.
- 2026-09-09 v0.29.0: Model Runner V2 becomes the default for all models, and ten deprecated architectures (Arctic, Chameleon, MPT and others) are removed. 594 commits from 277 contributors add Hy4-preview and Qwen3.8-Flash-Next support; `python -m vllm.entrypoints.openai.api_server` is deprecated in favour of `vllm serve`.
- 2026-08-26 v0.28.0: vLLM v0.28.0 lands 584 commits from 270 contributors: a Kimi K3 speed push (decode context parallelism, fused kernels, expert sharding saving ~17 GiB per GPU) plus end-to-end sparse MLA for DeepSeek V4.
- 2026-04-27 v0.20.0: 752 commits from 320 contributors: DeepSeek V4 support, FlashAttention 4 as the default MLA prefill backend, TurboQuant 2-bit KV cache for 4x capacity, and CUDA 13 plus PyTorch 2.11 as defaults.