Overview
DeepInfer is the inference framework behind DeepSeek's large-scale production services, published by DeepSeek under the Apache-2.0 license. The open-source release covers offline inference with the core features DeepSeek needs to serve its models at scale: optimized kernels, a model and sampler runtime, and the parallelism layouts used in its own clusters. It currently supports one model, DeepSeek V4.1 Flash, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
The framework is built primarily on vLLM, with ideas taken from SGLang and TensorRT-LLM, and it leans on DeepSeek's own kernel libraries — FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect — plus custom TileLang kernels. On top of that base it adds attention data parallelism, balanced context parallelism and MoE expert parallelism with expert load balancing (EPLB), DSpark speculative decoding with confidence-scheduled verification, SWA bounded replay to cut prefill work on cache hits, and an asynchronous batch scheduler that is on by default.
The repository also carries a long deployment Q&A that explains how DeepSeek runs the complete service: disaggregated vision-encoder, prefill and decode stages, wide expert parallelism for decode at EP64 or above, a dedicated 3FS storage cluster for the KV cache, autoscaling and fault handling, and the low-precision formats used for KV and expert weights. Several pieces of that full service are not in this release — a KV cache system, prefill/decode disaggregation and load balancing across data-parallel replicas — so DeepInfer today is an offline engine, not a drop-in serving stack.
What it does
- Optimized kernels, model and sampler runtime for DeepSeek V4.1 Flash
- Attention data parallelism, balanced context parallelism and MoE expert parallelism with EPLB
- DSpark speculative decoding with confidence-scheduled, variable-length verification of draft tokens
- SWA bounded replay, which recomputes only a short recent window on a cache hit to reduce prefill work
- Asynchronous, zero-overhead batch scheduling, enabled by default
- Batch-invariant inference and deterministic sampling on NVIDIA GPUs
- Structured outputs constrained by a regex, a JSON schema or an EBNF grammar
- Runs on NVIDIA SM100 GPUs (CUDA 13) and Huawei Ascend 950 NPUs (CANN 9.2)
Getting started
DeepInfer needs Linux, GCC 13 or newer and Python 3.12, plus CUDA 13 on NVIDIA or CANN 9.2 on Ascend. The steps below follow the project README.
Create an empty Python 3.12 environment
Start from a fresh Python 3.12 environment and activate it before building.
Choose the accelerator and build
Set the accelerator to `nvidia`, or to `ascend` on the Ascend platform, and keep it exported in the shell where you run DeepInfer.
export DEEPINFER_ACCELERATOR=nvidia
./build.shRun offline inference
The example downloads the default checkpoint, `deepseek-ai/DeepSeek-V4.1-Flash-1017`, to the Hugging Face cache on first run. Set `DEEPINFER_HF_MODEL_DIR` to use a local copy instead.
python examples/offline_inference.pyTurn on optional features
Features are switched with environment variables: `DEEPINFER_ENABLE_MTP_VARLEN=1` enables speculative decoding, and `DEEPINFER_BATCH_INVARIANT=1` enables batch-invariant inference on NVIDIA. Structured outputs are requested by passing `so_args` to `SamplingParams`.
DEEPINFER_ENABLE_MTP_VARLEN=1 python examples/offline_inference.pyCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Running DeepSeek V4.1 Flash open weights with the same engine DeepSeek uses in production
- Studying how a frontier lab combines expert parallelism, speculative decoding and cache replay at scale
- Reproducible evaluation runs that need batch-invariant inference or deterministic sampling
- Serving DeepSeek models on Huawei Ascend 950 NPUs as well as NVIDIA SM100 GPUs
- Generating JSON-schema, regex or grammar-constrained output from DeepSeek V4.1 Flash
How DeepInfer compares
DeepInfer alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Transformers | ★ 168k | Hugging Face Transformers is a Python framework that defines and runs state-of-the-art pretrained models for text, vision, audio, and multimodal tasks, for both inference and training. |
| vLLM | ★ 94.3k | A high-throughput LLM serving engine that uses PagedAttention and continuous batching to serve many requests at once. |
| SGLang | ★ 37.5k | A serving framework for LLMs and multimodal models that boosts throughput by reusing shared prompt prefixes across requests. |
| Modular Platform | ★ 30k | Modular's AI platform: the MAX inference framework with an OpenAI-compatible serving endpoint and GPU kernel library, plus the Mojo language and compiler. |
| TensorRT-LLM | ★ 14.9k | NVIDIA's library that compiles LLMs into optimized engines for the fastest inference on its data-center GPUs. |
| OpenLLM | ★ 12.6k | A tool to run any open-source LLM as an OpenAI-compatible API endpoint locally or in the cloud. |
| LMCache | ★ 12.1k | A KV-cache layer that stores and shares cached attention state across engines and requests to cut repeated computation. |
| DeepInfer | — | The high-performance inference framework that powers DeepSeek's own production services, released as open source |