Overview
SGLang-Omni is a serving framework from the sgl-project team, the people behind SGLang, aimed at a class of models that a plain LLM server handles poorly: text-to-speech, speech recognition and unified "omni" models that take text, images, video or audio in and produce text or audio out. These models do not generate in one pass. A single request flows through preprocessing, encoders, an autoregressive engine, a talker, a decoder or vocoder and an aggregator, each with its own compute pattern and resource needs. SGLang-Omni treats that chain as coordinated stages and owns the pipeline topology, stage lifecycle and the transport between stages.
Each stage runs behind a scheduler matched to its job — SGLang-backed autoregressive scheduling with KV cache, batching and prefill/decode scheduling where a model has an LLM core, lightweight schedulers for preprocessing, and streaming loops for vocoders. A control plane routes requests and signals between stages, while a separate relay data plane moves tensor payloads over shared memory, NCCL, NIXL or Mooncake. In other words, SGLang-Omni does not replace SGLang: it composes with it for the autoregressive parts and adds the multi-stage plumbing around them.
On the outside it is an OpenAI-compatible server. /v1/audio/speech covers single, batch and streaming speech generation plus uploaded voices and voice cloning; /v1/audio/transcriptions handles ASR, including speaker labels and timestamps for MOSS-Transcribe-Diarize; and chat endpoints serve omni models such as Qwen3-Omni and Ming-Omni. The model list spans Higgs Audio v3, Fish Speech S2-Pro, Qwen3-TTS, Voxtral TTS, MOSS-TTS, ZONOS2, Qwen3-ASR, Fun-ASR and MiniMax Music 3, each with a cookbook page. NVIDIA CUDA is the fully supported backend; Apple Silicon and Intel GPUs are marked experimental for a subset of models. It is Python, Apache-2.0, and published on PyPI as sglang-omni.
What it does
- Multi-stage runtime that models generation as preprocessing, encoder, autoregressive, talker, decoder, vocoder and aggregator stages, each with its own scheduler
- Reuses SGLang's core for the autoregressive stages (KV cache, batching, prefill/decode scheduling) and adds streaming vocoder loops and light preprocessing schedulers around it
- Separate control and data planes: requests and signals go through the coordinator, tensors move stage-to-stage over shared memory, NCCL, NIXL or Mooncake
- OpenAI-compatible endpoints for multimodal chat, speech generation (single, batch, streaming, uploaded voices) and transcription, plus a /v1/realtime WebSocket demo for live speech-to-text
- Per-model cookbooks for TTS, ASR, omni and music models, from Fish Speech S2-Pro and Qwen3-TTS to Qwen3-Omni and MiniMax Music 3
- SGLang-Omni Router: a multi-worker OpenAI-compatible front door with health, readiness, lifecycle and capability discovery
Getting started
The project recommends its Docker image on NVIDIA GPUs, since UCX, flash-attn, SGLang and CUDA come prebuilt. The steps below follow the installation and TTS guides; Apple Silicon, Intel XPU, CPU, MUSA and Ascend NPU have their own install pages.
Start the container
Only a dev tag is published, and it moves with main — pin by digest when you need reproducible runs.
docker pull hongccc/sglang-omni:dev
docker run -it \
--shm-size 32g \
--gpus all \
--ipc host \
--network host \
--privileged \
hongccc/sglang-omni:dev \
/bin/zshInstall sglang-omni inside it
pip install --upgrade pip
pip install uv
uv venv .venv -p 3.12
source .venv/bin/activate
uv pip install --prerelease=allow "sglang-omni==0.1.7"Serve a TTS model
The TTS guide uses Fish Speech S2-Pro (complete its model-specific DAC prerequisites from the cookbook first). The example config ships in the repo's examples/configs directory; the allowed-media-domain flags are only needed when requests fetch reference audio from Hugging Face.
hf download fishaudio/s2-pro
sgl-omni serve \
--model-path fishaudio/s2-pro \
--config examples/configs/s2pro_tts.yaml \
--allowed-media-domain huggingface.co \
--allowed-media-domain cas-bridge.xethub.hf.co \
--port 8000Generate speech through the OpenAI-style endpoint
curl -X POST http://localhost:8000/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "fishaudio/s2-pro",
"voice": "default",
"input": "Hello, how are you?"
}' \
--output output.wavOr transcribe audio with an ASR model
Speech recognition models such as Qwen3-ASR are served the same way and answer on /v1/audio/transcriptions, one uploaded file per request.
sgl-omni serve \
--model-path Qwen/Qwen3-ASR-1.7B \
--model-name Qwen/Qwen3-ASR-1.7B \
--port 8000
curl -X POST http://localhost:8000/v1/audio/transcriptions \
-F model=Qwen/Qwen3-ASR-1.7B \
-F file=@tests/data/query_to_cars.wav \
-F response_format=jsonCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Self-host open text-to-speech models — with voice cloning, batch and streaming output — behind the same API shape as OpenAI's speech endpoint
- Run open speech recognition, including speaker diarization with timestamps, as an OpenAI-compatible transcription service
- Serve an omni model such as Qwen3-Omni that takes audio, images or video in and answers in text or speech, including live microphone input over /v1/realtime
- Put several speech workers behind one front door with the SGLang-Omni Router for health checks and capability discovery
How SGLang-Omni compares
SGLang-Omni alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Transformers | ★ 168k | Hugging Face Transformers is a Python framework that defines and runs state-of-the-art pretrained models for text, vision, audio, and multimodal tasks, for both inference and training. |
| vLLM | ★ 93.8k | A high-throughput LLM serving engine that uses PagedAttention and continuous batching to serve many requests at once. |
| SGLang | ★ 37.1k | A serving framework for LLMs and multimodal models that boosts throughput by reusing shared prompt prefixes across requests. |
| Modular Platform | ★ 29.9k | Modular's AI platform: the MAX inference framework with an OpenAI-compatible serving endpoint and GPU kernel library, plus the Mojo language and compiler. |
| TensorRT-LLM | ★ 14.8k | NVIDIA's library that compiles LLMs into optimized engines for the fastest inference on its data-center GPUs. |
| OpenLLM | ★ 12.6k | A tool to run any open-source LLM as an OpenAI-compatible API endpoint locally or in the cloud. |
| LMCache | ★ 12.1k | A KV-cache layer that stores and shares cached attention state across engines and requests to cut repeated computation. |
| SGLang-Omni | ★ 1.5k | A multi-stage serving runtime for TTS, ASR and omni models, with OpenAI-compatible speech, transcription and multimodal chat endpoints |