RTP-LLM

// Overview

RTP-LLM is a large language model inference acceleration engine built by Alibaba's Foundation Model Inference Team and released under the Apache 2.0 licence. Inside Alibaba Group it serves LLM traffic for business units including Taobao, Tmall, Idlefish, Cainiao, Amap, Ele.me, AE and Lazada, and it is a sub-project of Alibaba's havenask search engine. The project credits NVIDIA's FasterTransformer as its main base, with some kernels integrated from TensorRT-LLM and ideas drawn from vLLM and Hugging Face Transformers.

As a single-node engine it loads Hugging Face models in SafeTensors, PyTorch or Megatron formats and runs them on high-performance CUDA kernels such as PagedAttention, FlashAttention and FlashDecoding. It quantizes weights to INT8 automatically at load time, accepts GPTQ and AWQ INT4 checkpoints, can quantize the KV cache, serves several LoRA adapters from one model instance, and handles image-plus-text input. The scheduling and batching layer is written in C++, and a running server answers OpenAI-style /v1/chat/completions requests as well as its own native generation endpoint. The documentation lists NVIDIA GPUs of compute capability 7.0 and up (V100 and T4 through H100, H200 and H20) and AMD's MI308X.

The 0.2.0 release turned it into a distributed serving system for very large mixture-of-experts models such as DeepSeek-V3/R1, Kimi-K2 and Qwen3: prefill/decode disaggregation, attention/FFN disaggregation, a separately deployable frontend, expert-parallel load balancing (EPLB), DeepEP communication, micro-batch overlapping and MTP speculative decoding. A load-balancing Master role, FlexLB, routes each request to prefill nodes by KV-cache hit rate and queue time and to decode nodes by KV-cache usage, with a cache server for prefix-aware routing across clusters of 100 to 1,000 nodes.

// What it does

• Optimized CUDA kernels — PagedAttention, FlashAttention, FlashDecoding, plus XQA and FlashInfer attention backends — with framework-level tuning of dynamic batching overhead
• Quantization built in: automatic weight-only INT8 at load time, GPTQ and AWQ INT4 checkpoints, adaptive KV-cache quantization and an FP8 KV cache
• Multi-LoRA serving from a single model instance, multimodal image-and-text input, P-tuning models and pruned irregular models
• Prefix cache for multi-turn dialogue, system-prompt cache, KV-cache reuse and speculative decoding, including MTP

• Distributed deployment: tensor parallelism across GPUs and machines, prefill/decode and attention/FFN disaggregation, expert parallelism with EPLB and DeepEP, and the FlexLB load balancer
• Runs on NVIDIA GPUs from compute capability 7.0 (with specific V100 optimizations) and on AMD MI308X, with Docker images and wheels per card type

// Getting started

RTP-LLM ships as a Python wheel and as Docker images per GPU type. The commands below follow the project's install and quick-start pages; swap in your own checkpoint path and model type.

pip install --upgrade pip
pip install "rtp_llm>=0.2.0"

docker run --gpus all --shm-size 32g -p 30000:30000 -v /mnt:/mnt -v /home:/home --ipc=host \
  ali-hangzhou-hub-registry.cn-hangzhou.cr.aliyuncs.com/isearch/rtp_llm_sm8x_opensource:0.2.0_0.2.0_2025_10_09_17_35_8fa289f5 \
  /opt/conda310/bin/python -m rtp_llm.start_server \
  --checkpoint_path=/mnt/nas1/hf/models--Qwen--Qwen1.5-0.5B-Chat/snapshots/6114e9c18dac0042fa90925f03b046734369472f/ \
  --model_type=qwen_2 --start_port=30000

python -m rtp_llm.start_server \
  --checkpoint_path=/path/to/hf/checkpoint/ \
  --model_type=qwen_2 \
  --start_port=8090

curl -s http://localhost:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "What is the capital of France?"}]}'

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// How RTP-LLM compares

RTP-LLM alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.