FastDeploy

// Overview

FastDeploy is the inference and deployment toolkit for large language models and vision-language models built on PaddlePaddle. It is aimed at production serving rather than experimentation: you start a server process, it exposes an OpenAI-compatible HTTP API on one port and a Prometheus metrics endpoint on another, and clients talk to it with the same request shapes they would use against any OpenAI-style endpoint.

Its headline capability is load-balanced PD disaggregation — running prefill and decode as separate, dynamically re-rolled instances with context caching, so resource use can be tuned against latency targets and throughput at the same time. KV cache movement between those instances goes through a lightweight transport library that picks NVLink or RDMA automatically. On top of that sit speculative decoding, Multi-Token Prediction and chunked prefill, plus quantization in W8A16, W8A8, W4A16, W4A8, W2A16 and FP8 formats.

FastDeploy is notable for breadth of silicon support: besides NVIDIA GPUs it targets Kunlunxin XPU, Hygon DCU, Iluvatar GPU, Enflame GCU, MetaX GPU and Intel Gaudi. It runs on Linux with Python 3.10–3.12, and is distributed as prebuilt Docker images and as a wheel installed from PaddlePaddle's own package index — not from PyPI.

// What it does

• Load-balanced prefill/decode disaggregation with context caching and dynamic instance role switching
• Unified KV cache transmission library that selects NVLink or RDMA automatically
• OpenAI-compatible API server with vLLM interface compatibility, launched by one command
• Quantization across W8A16, W8A8, W4A16, W4A8, W2A16 and FP8
• Speculative decoding, Multi-Token Prediction (MTP) and chunked prefill
• Multi-hardware backends: NVIDIA GPU, Kunlunxin XPU, Hygon DCU, Iluvatar GPU, Enflame GCU, MetaX GPU and Intel Gaudi

// Getting started

Requirements for the NVIDIA path are GPU driver ≥ 535, CUDA ≥ 12.3, cuDNN ≥ 9.5, Linux x86_64 and Python ≥ 3.10. Docker is the recommended route; the pip route installs paddlepaddle-gpu first and then fastdeploy-gpu from PaddlePaddle's index.

# CUDA 12.6
docker pull ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/fastdeploy-cuda-12.6:2.5.0
# CUDA 12.9
docker pull ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/fastdeploy-cuda-12.9:2.5.0

python -m pip install paddlepaddle-gpu==3.3.1 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
python -m pip install fastdeploy-gpu==2.5.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/ --extra-index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

export ENABLE_V1_KVCACHE_SCHEDULER=1
python -m fastdeploy.entrypoints.openai.api_server \
       --model baidu/ERNIE-4.5-0.3B-Paddle \
       --port 8180 \
       --metrics-port 8181 \
       --engine-worker-queue-port 8182 \
       --max-model-len 32768 \
       --max-num-seqs 32

curl -i http://0.0.0.0:8180/health

curl -X POST "http://0.0.0.0:8180/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
  "messages": [
    {"role": "user", "content": "Write me a poem about large language model."}
  ],
  "stream": true
}'

import openai
host = "0.0.0.0"
port = "8180"
client = openai.Client(base_url=f"http://{host}:{port}/v1", api_key="null")

response = client.chat.completions.create(
    model="null",
    messages=[
        {"role": "system", "content": "I'm a helpful AI assistant."},
        {"role": "user", "content": "Write me a poem about large language model."},
    ],
    stream=True,
)
for chunk in response:
    if chunk.choices[0].delta:
        print(chunk.choices[0].delta.content, end='')

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Serving ERNIE and other PaddlePaddle-format LLMs and VLMs behind an OpenAI-compatible endpoint
• Splitting prefill and decode across separate instances to hit latency targets without giving up throughput
• Deploying LLM inference on non-NVIDIA silicon — Kunlunxin, Hygon, Iluvatar, Enflame, MetaX or Intel Gaudi
• Squeezing a large model onto fewer cards with W4A8, W2A16 or FP8 quantization plus speculative decoding

// How FastDeploy compares

FastDeploy alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.