GPUStack

// Overview

GPUStack is an open-source GPU cluster manager for deploying AI models. It configures and orchestrates inference engines such as vLLM, SGLang, and TensorRT-LLM, and schedules work across the GPUs in your clusters. You can also plug in your own engine when the built-in ones do not fit.

It is aimed at development teams, IT organizations, and service providers that need to run model serving at scale rather than on a single machine. A GPUStack server can manage GPU clusters spread across on-premises servers, Kubernetes, and cloud providers, and exposes industry-standard APIs for LLM, voice, image, and video models.

Within the inference and serving category, GPUStack sits one layer above a single serving engine: instead of running one vLLM process by hand, you let GPUStack pick engines, tune parameters, allocate GPUs, and add operational features like authentication, load balancing, monitoring, and usage metering.

// What it does

• Manages GPU clusters across on-premises servers, Kubernetes, and cloud providers from one server
• Automatically configures inference engines including vLLM, SGLang, and TensorRT-LLM, with support for custom engines
• Pre-tuned modes for low latency or high throughput, plus KV cache systems (LMCache, HiCache) and speculative decoding (EAGLE3, MTP, N-grams)
• Built-in user authentication, access control, load balancing, and automated failure recovery
• Real-time GPU monitoring and token/request metering via integrated Grafana and Prometheus dashboards
• Supports a wide range of accelerators: NVIDIA, AMD, Ascend NPU, Hygon DCU, MThreads, Iluvatar, MetaX, Cambricon MLU, and T-Head PPU

// Getting started

Install the GPUStack server with Docker, log into the UI, then add a GPU cluster and worker nodes. Worker nodes require Linux, an NVIDIA driver, Docker, and the NVIDIA Container Toolkit.

sudo docker run -d --name gpustack \
    --restart unless-stopped \
    -p 80:80 \
    --volume gpustack-data:/var/lib/gpustack \
    gpustack/gpustack

sudo docker exec gpustack cat /var/lib/gpustack/initial_admin_password

sudo docker run -d --name gpustack-worker \
      --restart=unless-stopped \
      --privileged \
      --network=host \
      --volume /var/run/docker.sock:/var/run/docker.sock \
      --volume gpustack-data:/var/lib/gpustack \
      --runtime nvidia \
      gpustack/gpustack \
      --server-url http://your_gpustack_server_url \
      --token your_worker_token \
      --advertise-address 192.168.1.2

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Pooling GPUs from several on-prem and cloud machines into one cluster to serve models behind a single API
• Running Model-as-a-Service for internal teams with authentication, access control, and per-user token metering
• Deploying new models on release day by relying on pluggable engines like vLLM, SGLang, or TensorRT-LLM
• Tuning serving for low latency or high throughput using KV cache and speculative decoding options without hand-configuring each engine

// How GPUStack compares

GPUStack alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.