vLLM Ascend

// Overview

vLLM Ascend (`vllm-ascend`) is the vLLM project's own hardware plugin for running vLLM on Huawei Ascend NPUs. It lives under the `vllm-project` organisation and is community maintained; the vLLM community names it the recommended way to support the Ascend backend. Rather than maintaining a fork, it implements the hardware-pluggable interface described in vLLM's "Hardware pluggable" RFC, so the Ascend integration is decoupled from vLLM's own tree.

The practical consequence is that vLLM's API does not change. Install the plugin alongside a matching vLLM version and the same `LLM(...)` calls and `vllm serve` command work against Ascend hardware, with transformer-style models, Mixture-of-Experts models, embedding models and multimodal LLMs all supported. The project maintains a support matrix listing exactly which models and features are covered on which devices.

Versioning is tied to vLLM's. The plugin's own version tracks the vLLM release it targets — the README asks for "vLLM (the same version as vllm-ascend)" — and release branches are cut alongside vLLM releases (`releases/v0.23.0` for vLLM 0.23.0, and so on), while `main` tracks vLLM's main branch under Ascend CI. Several release branches are maintained in parallel, each with its own CI commitment.

Hardware and software requirements are specific: Atlas 800I A2 and 800I A3 inference series, Atlas A2 and A3 training series, and Atlas 300I Duo experimentally; Linux, Python 3.10–3.12, CANN 9.1.0, and PyTorch 2.10.0 with TorchNPU 2.10.0.post4. Prebuilt container images are published on Quay, and Huawei's HiDevLab offers online NPU access for development and testing.

// What it does

• A hardware plugin, not a fork — implements vLLM's pluggable hardware interface so Ascend support ships out of tree
• Runs transformer-like, Mixture-of-Experts, embedding and multimodal models with the ordinary vLLM Python API and `vllm serve`
• A published support matrix stating which models and features work on which Ascend devices
• Release branches cut alongside vLLM releases, several maintained in parallel with their own Ascend CI commitments
• Prebuilt container images on Quay (`quay.io/ascend/vllm-ascend`) covering the driver mounts an NPU container needs
• Documented large-scale Expert Parallelism deployment, and user stories covering LLaMA-Factory, verl, TRL and GPUStack

// Getting started

The container image is the shortest path: it carries the plugin, a matching vLLM and the Ascend runtime, and needs the NPU device and driver paths mounted in.

export DEVICE=/dev/davinci0
export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0
docker run --rm --name vllm-ascend --shm-size=1g --device $DEVICE \
--device /dev/davinci_manager --device /dev/devmm_svm \
--device /dev/hisi_hdc -v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache -p 8000:8000 -it $IMAGE bash

from vllm import LLM, SamplingParams

prompts = ["Hello, my name is", "The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="Qwen/Qwen3-0.6B")
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

vllm serve Qwen/Qwen3-0.6B &

curl http://localhost:8000/v1/completions -H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen3-0.6B", "prompt": "Beijing is a", \
"max_completion_tokens": 5, "temperature": 0}'

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when the inference hardware is Huawei Ascend and you want vLLM's scheduler, paged attention and OpenAI-compatible server rather than a different stack
• Reach for it to keep one serving codebase across GPU and NPU fleets, since the plugin leaves the vLLM API untouched
• Reach for it for large-scale Expert Parallelism deployments of MoE models on Ascend clusters
• Reach for it when a fine-tuning or RL stack (LLaMA-Factory, verl, TRL) needs a vLLM-backed rollout engine on Ascend

// How vLLM Ascend compares

vLLM Ascend alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.