█

AI/TLDR

Xinference

Serve LLM, embedding, and multimodal models behind one OpenAI-compatible API

GitHub★ 9.6kWebsiteinference.readthedocs.io
Serving & DeploymentOpen source
Language
Python
$pip install "xinference[all]"

Overview

Xinference (Xorbits Inference) is an open-source platform for serving language, speech, embedding, and multimodal models. You launch a built-in or custom model with a single command, and it exposes that model over an OpenAI-compatible REST API, plus RPC, a CLI, and a web UI.

It is aimed at developers and data teams who want to run open models on their own hardware instead of calling a hosted API. Under the hood it can use different inference engines, including vLLM, llama.cpp, and Hugging Face Transformers, and it can spread work across GPUs and CPUs on one machine or several.

As a model-serving tool, it sits between your models and your application code. Because the API matches the OpenAI format, existing clients and libraries such as LangChain, LlamaIndex, and Dify can talk to it with little change.

What it does

  • OpenAI-compatible RESTful API, including a Function Calling API, plus RPC, CLI, and a web UI for managing models
  • Multiple inference backends: vLLM, llama.cpp (via xllamacpp), Transformers, SGLang, and MLX
  • Built-in catalog of open models covering chat, embeddings, speech, and multimodal tasks, each launchable with one command
  • Runs on heterogeneous hardware, using GPUs and CPUs, with CPU and Metal support
  • Distributed deployment that spreads model inference across multiple devices or machines
  • Integrations with LangChain, LlamaIndex, Dify, FastGPT, RAGFlow, and Chatbox

Getting started

Install Xinference, start a local server, then launch a model and call it from the Python client.

Install Xinference

Install the full package with all backends. You can also install a single backend extra such as xinference[vllm], xinference[llama_cpp], or xinference[transformers].

bashbash
pip install "xinference[all]"

Start the local server

Run the local server, which serves the API and web UI on the chosen host and port.

bashbash
xinference-local --host 0.0.0.0 --port 9997

Launch a model

Launch a built-in model from the CLI, choosing the inference engine, name, size, and format.

bashbash
xinference launch --model-engine vllm -n qwen2.5-instruct -s 0_5 -f pytorch

Call the model from Python

Connect with the RESTful client, get the running model, and send a chat message.

pythonpython
from xinference.client import RESTfulClient

client = RESTfulClient("http://127.0.0.1:9997")
model = client.get_model("qwen2.5-instruct")
model.chat(
    messages=[
        {"role": "user", "content": "Who won the world series in 2020?"}
    ]
)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Self-host open chat models behind an OpenAI-compatible endpoint so existing OpenAI client code keeps working
  • Serve embedding models for a RAG pipeline alongside chat and reranking models from one platform
  • Run inference across several GPUs or machines for larger models or higher throughput
  • Connect a serving backend to tools like Dify, LangChain, or LlamaIndex without writing a custom API layer

How Xinference compares

Xinference alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
Transformers★ 168kHugging Face Transformers is a Python framework that defines and runs state-of-the-art pretrained models for text, vision, audio, and multimodal tasks, for both inference and training.
vLLM★ 95.2kA high-throughput LLM serving engine that uses PagedAttention and continuous batching to serve many requests at once.
SGLang★ 37.6kA serving framework for LLMs and multimodal models that boosts throughput by reusing shared prompt prefixes across requests.
Modular Platform★ 30kModular's AI platform: the MAX inference framework with an OpenAI-compatible serving endpoint and GPU kernel library, plus the Mojo language and compiler.
DeepInfer★ 20kDeepSeek's own large-scale inference framework for its models, with expert parallelism, confidence-scheduled speculative decoding and NVIDIA plus Ascend support.
TensorRT-LLM★ 14.9kNVIDIA's library that compiles LLMs into optimized engines for the fastest inference on its data-center GPUs.
OpenLLM★ 12.6kA tool to run any open-source LLM as an OpenAI-compatible API endpoint locally or in the cloud.
Xinference★ 9.6kServe LLM, embedding, and multimodal models behind one OpenAI-compatible API