LMDeploy

// Overview

LMDeploy is an open-source toolkit for compressing, deploying, and serving large language models. It is developed by the MMRazor and MMDeploy teams and is built around two inference backends: TurboMind, a C++/CUDA engine, and a pure-Python PyTorch engine that is easier to extend.

It is aimed at engineers who need to run open models such as Llama, Qwen, InternLM, DeepSeek, and many vision-language models in production rather than just in a notebook. You can load Hugging Face models directly, quantize them to lower precision, and expose them over an OpenAI-compatible HTTP API.

Within the high-throughput serving category, LMDeploy focuses on combining quantization (weight-only and KV cache) with serving features like continuous batching and a blocked KV cache, so the same toolkit handles both shrinking the model and serving requests at scale.

// What it does

• Two inference engines: the C++ TurboMind engine and a pure-Python PyTorch engine for easier experimentation
• Request throughput up to 1.8x higher than vLLM via persistent (continuous) batching, blocked KV cache, dynamic split and fuse, and tensor parallelism
• Weight-only and KV cache quantization, including 4-bit AWQ inference reported at 2.4x faster than FP16
• OpenAI-compatible API server so existing OpenAI client code can talk to local models
• Support for many LLMs and vision-language models (Llama, Qwen, InternLM, DeepSeek, InternVL, and more), loadable directly from Hugging Face
• Multi-model, multi-machine, multi-card serving through a request distribution service

// Getting started

Install LMDeploy from PyPI, run a quick offline inference check, then launch the API server when you are ready to serve over HTTP. Prebuilt wheels from v0.13.0 onward are built against CUDA 12.8, and Python 3.10-3.13 is supported.

pip install lmdeploy

import lmdeploy
with lmdeploy.pipeline("internlm/internlm3-8b-instruct") as pipe:
    response = pipe(["Hi, pls intro yourself", "Shanghai is"])
    print(response)

lmdeploy serve api_server internlm/internlm2_5-7b-chat

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Serving an open LLM or VLM behind an OpenAI-compatible endpoint so existing client code can call a self-hosted model
• Quantizing a model to 4-bit (AWQ) or applying KV cache quantization to fit it on smaller or fewer GPUs
• Running high-throughput batch inference where continuous batching and a blocked KV cache improve requests per second
• Deploying multi-model services across multiple machines and GPUs through the request distribution service

// How LMDeploy compares

LMDeploy alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.