Chitu

// Overview

Chitu (赤兔) is an open-source inference framework for large language models from the PACMAN group at Tsinghua University. It serves the same role as vLLM or SGLang — take an open-weight checkpoint, serve it behind an HTTP API with paged KV cache, tensor and pipeline parallelism and CUDA graphs — but with a much wider hardware target than most engines attempt.

That breadth is the point of the project. Alongside NVIDIA GPUs from flagship to older generations, it supports Huawei Ascend, Moore Threads, Muxi and Haiguang accelerators, and it can run CPU-only or in mixed CPU+GPU configurations. The model list covers the frontier Chinese open-weight families — DeepSeek (including DeepSeek-R1 671B), Qwen, GLM and Kimi — with the full matrix maintained in the repository's SUPPORTED_MODELS document.

Deployment comes in two shapes. Built from source it is a Python package launched through `torchrun`, with configuration passed as dotted key-value overrides on the command line. For production the project also publishes `chitu.run`, a single executable on the releases page that covers multi-node, multi-instance and prefill/decode-disaggregated topologies without a build step. The server exposes OpenAI-compatible `/v1/chat/completions` and `/v1/completions` endpoints as well as an Anthropic-compatible `/v1/messages`. The project is Apache-2.0, with third-party snippets marked by SPDX identifiers in `LICENSES/` and submodules keeping their own terms.

// What it does

• Runs on NVIDIA GPUs across several generations, plus Ascend, Moore Threads, Muxi and Haiguang accelerators
• CPU-only and hybrid CPU+GPU deployment as well as single-GPU and multi-node clusters
• Paged KV cache, tensor and pipeline parallelism, CUDA graphs and MLA absorption options exposed as launch-time config keys
• Serves DeepSeek (including R1 671B), Qwen, GLM and Kimi checkpoints
• OpenAI-compatible chat and completion endpoints plus an Anthropic-compatible messages endpoint on the same port
• A single `chitu.run` executable for multi-node, multi-instance and prefill/decode-disaggregated deployments

// Getting started

Two routes: download the packaged `chitu.run` executable from the releases page, or build from source. The source build needs the submodules, so clone recursively.

git clone --recursive https://github.com/thu-pacman/chitu && cd chitu

pip install -r requirements-build.txt
pip install -U torch --index-url https://download.pytorch.org/whl/cu124

TORCH_CUDA_ARCH_LIST=9.0 pip install --no-build-isolation . \
  -c <(pip list --format freeze | grep -v "flash-mla" | grep -v "flash_mla")

export WORLD_SIZE=8
torchrun --nnodes 1 \
    --nproc_per_node 8 \
    --master_port=22525 \
    -m chitu \
    serve.port=21002 \
    infer.cache_type=paged \
    infer.pp_size=1 \
    infer.tp_size=8 \
    models=DeepSeek-R1 \
    models.ckpt_dir=/data/DeepSeek-R1 \
    infer.mla_absorb=absorb-without-precomp \
    infer.raise_lower_bit_float_to=bfloat16 \
    infer.max_batch_size=1 \
    infer.max_seq_len=4096 \
    infer.use_cuda_graph=True

curl localhost:21002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Serving a 671B-class open model on a cluster that is not all NVIDIA
• Standing up an OpenAI-compatible endpoint for DeepSeek, Qwen, GLM or Kimi checkpoints you host yourself
• Running inference where CPU capacity has to make up for limited accelerator memory
• Deploying multi-node or prefill/decode-disaggregated topologies from one executable rather than a built environment

// How Chitu compares

Chitu alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.