SGLang

// Overview

SGLang is a serving framework for large language models and multimodal models. It runs your model behind an HTTP server and answers requests through an OpenAI-compatible API, so existing clients can talk to it with little change. It is built to give low-latency, high-throughput inference from a single GPU up to large distributed clusters.

Its main idea is to reuse work across requests. RadixAttention caches shared prompt prefixes, so when many requests start with the same system prompt or few-shot examples, the engine does not recompute that part each time. Combined with continuous batching and a low-overhead scheduler, this raises the number of tokens a server can produce per second.

It fits the high-throughput serving category alongside engines you put in front of a model in production. It supports a wide range of language models (Llama, Qwen, DeepSeek, GLM, Gemma, Mistral and more), embedding models, reward models, and diffusion models, and offers quantization, tensor and pipeline parallelism, structured outputs, and multi-LoRA batching.

// What it does

• RadixAttention prefix caching reuses shared prompt prefixes across requests to cut repeated computation
• OpenAI-compatible HTTP server, so existing OpenAI client code can point at your local endpoint
• Continuous batching, paged attention, chunked prefill, and a zero-overhead CPU scheduler for higher throughput
• Tensor, pipeline, expert, and data parallelism to scale from one GPU to multi-GPU clusters
• Quantization support (FP8, FP4, INT4, AWQ, GPTQ) plus multi-LoRA batching
• Broad model coverage: language, embedding, reward, and diffusion models including Llama, Qwen, DeepSeek, GLM, Gemma, and Mistral

// Getting started

Install SGLang, launch a server for a model, then send requests to its OpenAI-compatible endpoint on port 30000.

pip install --upgrade pip
pip install uv
uv pip install sglang

python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
  --host 0.0.0.0 --log-level warning

curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen/qwen2.5-0.5b-instruct", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Serve an open model behind an OpenAI-compatible API so existing app code works without rewriting client calls
• Run chat or agent workloads where many requests share a long system prompt, letting prefix caching skip repeated work
• Scale inference across multiple GPUs with tensor or pipeline parallelism for large models like DeepSeek or Qwen
• Serve multiple LoRA adapters in one batch, or run quantized models to fit larger workloads on limited GPU memory

// Version history

Every verified update to SGLang that AI/TLDR tracked, newest first — each links to our coverage and the official changeset.

1. 2026-09-18v0.5.20

   SGLang v0.5.20 lands 713 pull requests from 237 contributors and stops publishing CUDA 12 wheels and images. The unified radix tree now caches branching points, lifting the token hit rate on DeepSeek-V4-Flash from 43.8% to 60.8%, and Mamba 1 and Mamba 2 state-space models become servable.

   our coverage →changeset ↗
2. 2026-09-05v0.5.19

   SGLang v0.5.19 adds beam search: pass beam_width and the server returns the n best sequences instead of one sample. The release lands 786 pull requests from 214 contributors, makes the unified radix tree the prefix cache for every model, and adds a Lean attention kernel that raises MI355X throughput by up to 1.52x.

   our coverage →changeset ↗
3. 2026-08-22v0.5.18

   SGLang v0.5.18 stages model weights from storage while CUDA graphs capture, cutting a Qwen3-32B cold start on an H100 from 84.8 to 35.6 seconds, and adds serving support for seven new model families.

   our coverage →changeset ↗

// SGLang in the news

1. 2026-10-29MAJORMistral Large 4 open weights — Le Chonk lands on Hugging Face
2. 2026-10-26MAJORCloudflare GLM-5.3 leak — KV cache bug sent 155 replies to the wrong customers
3. 2026-10-13MAJORFish Audio Drama 3 — an open 35B speech model for multi-character scenes
4. 2026-10-09MAJORClef-omni — Cloudflare's open decision model now reads audio and video
5. 2026-09-18MAJORSGLang v0.5.20 — CUDA 12 wheels retired, radix cache covers every model
6. 2026-09-08MAJORMiles v0.1 — RadixArk publishes the technical report for its open RL stack
7. 2026-09-05MAJORSGLang v0.5.19 — beam search arrives, plus 786 merged pull requests
8. 2026-08-31MAJORDeepSeek-V4-Flash-Vision-Exp weights go public — 305B multimodal MoE under MIT

From the AI/TLDR release feed — every item is source-verified when it ships.

// How SGLang compares

SGLang alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.