█

AI/TLDR

llm-d

Kubernetes-native distributed LLM inference with disaggregated serving and cache-aware routing

github.com/llm-d/llm-d★ 3.9kllm-d.ai

Overview

llm-d is a distributed inference serving stack for running open large language models on Kubernetes. It sits above model servers like vLLM and SGLang, adding orchestration, request routing, and KV-cache management so you can serve real production traffic across different hardware accelerators.

It is built for platform and ML infrastructure teams who already run on Kubernetes and need to serve high-scale traffic reliably rather than just run a model on one node. It ships well-lit path guides with benchmarked recipes and Helm charts so teams can start from a tested baseline instead of tuning everything by hand.

As a high-throughput serving engine, llm-d focuses on the layer between the model server and the cluster: intelligent routing, prefill/decode disaggregation, tiered KV-cache offloading, and SLO-aware autoscaling. It is a CNCF sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.

What it does

  • Intelligent routing with prefix-cache and load-aware balancing, plus experimental predicted-latency scheduling to cut latency and raise throughput
  • Tiered KV-cache management that offloads to CPU or disk and keeps a global index of cache state for multi-turn requests
  • Prefill/decode disaggregation and wide expert-parallelism for serving large models such as DeepSeek-R1 and GPT-OSS
  • SLO-aware autoscaling and flow control for stable multi-tenant production serving
  • Experimental OpenAI-compatible Batch APIs with asynchronous processing for large offline inference jobs
  • Built on open standards, integrating vLLM, SGLang, and Kubernetes Gateway API with Helm-based well-lit path recipes

Getting started

llm-d deploys onto a Kubernetes cluster using Helm and kubectl. The quickstart deploys the standalone router and an optimized-baseline model server; you need kubectl, Helm, and jq installed and a cluster already configured.

Clone the repo and set environment variables

Clone llm-d and export the versions and namespace used by the quickstart guide.

bashbash
git clone https://github.com/llm-d/llm-d.git && cd llm-d
export GAIE_VERSION=v1.5.0
export GUIDE_NAME="quickstart"
export NAMESPACE=llm-d-quickstart

Install CRDs and create the namespace

Apply the Gateway API Inference Extension CRDs and create a namespace for the deployment.

bashbash
kubectl apply -k "https://github.com/kubernetes-sigs/gateway-api-inference-extension/config/crd?ref=${GAIE_VERSION}"
kubectl create namespace ${NAMESPACE}

Deploy the router

Install the standalone scheduler chart with the optimized-baseline values.

bashbash
helm install ${GUIDE_NAME} \
    oci://registry.k8s.io/gateway-api-inference-extension/charts/standalone \
    -f guides/recipes/scheduler/base.values.yaml \
    -f guides/optimized-baseline/scheduler/optimized-baseline.values.yaml \
    -n ${NAMESPACE} --version ${GAIE_VERSION}

Deploy the model server

Apply the optimized-baseline vLLM model server into the namespace. See the official quickstart guide for validation and benchmarking steps.

bashbash
kubectl apply -n ${NAMESPACE} -k guides/optimized-baseline/modelserver/gpu/vllm/

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Serving a large open model like DeepSeek-R1 or GPT-OSS across many GPUs where prefill/decode disaggregation and expert-parallelism are needed
  • Running a multi-tenant LLM inference service on Kubernetes that needs SLO-aware autoscaling and stable behavior under load
  • Improving throughput and time-to-first-token on multi-turn chat workloads through prefix-cache-aware routing and KV-cache offloading
  • Standing up a benchmarked production baseline on a Kubernetes cluster instead of hand-tuning vLLM deployments from scratch

How llm-d compares

llm-d alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
Transformers★ 163kHugging Face Transformers is a Python framework that defines and runs state-of-the-art pretrained models for text, vision, audio, and multimodal tasks, for both inference and training.
vLLM★ 86.8kA high-throughput LLM serving engine that uses PagedAttention and continuous batching to serve many requests at once.
SGLang★ 30.6kA serving framework for LLMs and multimodal models that boosts throughput by reusing shared prompt prefixes across requests.
TensorRT-LLM★ 14.2kNVIDIA's library that compiles LLMs into optimized engines for the fastest inference on its data-center GPUs.
OpenLLM★ 12.4kA tool to run any open-source LLM as an OpenAI-compatible API endpoint locally or in the cloud.
NVIDIA Triton Inference Server★ 10.9kA multi-framework model server that runs TensorRT, PyTorch, ONNX, and other models with dynamic batching and concurrent execution.
LMCache★ 10.8kA KV-cache layer that stores and shares cached attention state across engines and requests to cut repeated computation.
llm-d★ 3.9kKubernetes-native distributed LLM inference with disaggregated serving and cache-aware routing