# SGLang

> Fast serving for LLMs and multimodal models with prefix-cache reuse

- Category: Models & Inference / Serving & Deployment
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/sgl-project/sglang
- Website: https://docs.sglang.ai
- Canonical page: https://ai-tldr.dev/tools/sglang/
- Updated: 2026-09-18

## In short

SGLang is a serving framework for large language models and multimodal models. It runs your model behind an HTTP server and answers requests through an OpenAI-compatible API, so existing clients can talk to it with little change. It sits in the Serving & Deployment category of the Models & Inference tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

SGLang is a serving framework for large language models and multimodal models. It runs your model behind an HTTP server and answers requests through an OpenAI-compatible API, so existing clients can talk to it with little change. It is built to give low-latency, high-throughput inference from a single GPU up to large distributed clusters.

Its main idea is to reuse work across requests. RadixAttention caches shared prompt prefixes, so when many requests start with the same system prompt or few-shot examples, the engine does not recompute that part each time. Combined with continuous batching and a low-overhead scheduler, this raises the number of tokens a server can produce per second.

It fits the high-throughput serving category alongside engines you put in front of a model in production. It supports a wide range of language models (Llama, Qwen, DeepSeek, GLM, Gemma, Mistral and more), embedding models, reward models, and diffusion models, and offers quantization, tensor and pipeline parallelism, structured outputs, and multi-LoRA batching.

## What it does

- RadixAttention prefix caching reuses shared prompt prefixes across requests to cut repeated computation
- OpenAI-compatible HTTP server, so existing OpenAI client code can point at your local endpoint
- Continuous batching, paged attention, chunked prefill, and a zero-overhead CPU scheduler for higher throughput
- Tensor, pipeline, expert, and data parallelism to scale from one GPU to multi-GPU clusters
- Quantization support (FP8, FP4, INT4, AWQ, GPTQ) plus multi-LoRA batching
- Broad model coverage: language, embedding, reward, and diffusion models including Llama, Qwen, DeepSeek, GLM, Gemma, and Mistral

## Getting started

Install SGLang, launch a server for a model, then send requests to its OpenAI-compatible endpoint on port 30000.

1. Install SGLang

   The docs recommend installing with uv for a faster setup.

```bash
pip install --upgrade pip
pip install uv
uv pip install sglang
```

2. Launch the server

   Start the server with a model path. By default it listens on port 30000.

```bash
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
  --host 0.0.0.0 --log-level warning
```

3. Send a request

   Query the OpenAI-compatible chat endpoint with curl. The base URL is http://localhost:30000/v1.

```bash
curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen/qwen2.5-0.5b-instruct", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
```

## When to use it

- Serve an open model behind an OpenAI-compatible API so existing app code works without rewriting client calls
- Run chat or agent workloads where many requests share a long system prompt, letting prefix caching skip repeated work
- Scale inference across multiple GPUs with tensor or pipeline parallelism for large models like DeepSeek or Qwen
- Serve multiple LoRA adapters in one batch, or run quantized models to fit larger workloads on limited GPU memory

## Version history

- 2026-09-18 v0.5.20: SGLang v0.5.20 lands 713 pull requests from 237 contributors and stops publishing CUDA 12 wheels and images. The unified radix tree now caches branching points, lifting the token hit rate on DeepSeek-V4-Flash from 43.8% to 60.8%, and Mamba 1 and Mamba 2 state-space models become servable.
- 2026-09-05 v0.5.19: SGLang v0.5.19 adds beam search: pass beam_width and the server returns the n best sequences instead of one sample. The release lands 786 pull requests from 214 contributors, makes the unified radix tree the prefix cache for every model, and adds a Lean attention kernel that raises MI355X throughput by up to 1.52x.
- 2026-08-22 v0.5.18: SGLang v0.5.18 stages model weights from storage while CUDA graphs capture, cutting a Qwen3-32B cold start on an H100 from 84.8 to 35.6 seconds, and adds serving support for seven new model families.