# GPUStack

> Pool your GPU machines into one cluster for serving AI models

- Category: Models & Inference / Serving & Deployment
- Repository: https://github.com/gpustack/gpustack
- Website: https://gpustack.ai
- Canonical page: https://ai-tldr.dev/tools/gpustack/

## In short

GPUStack is an open-source GPU cluster manager for deploying AI models. It configures and orchestrates inference engines such as vLLM, SGLang, and TensorRT-LLM, and schedules work across the GPUs in your clusters. It sits in the Serving & Deployment category of the Models & Inference tools. To get started: install and start the GPUStack server.

## Overview

GPUStack is an open-source GPU cluster manager for deploying AI models. It configures and orchestrates inference engines such as vLLM, SGLang, and TensorRT-LLM, and schedules work across the GPUs in your clusters. You can also plug in your own engine when the built-in ones do not fit.

It is aimed at development teams, IT organizations, and service providers that need to run model serving at scale rather than on a single machine. A GPUStack server can manage GPU clusters spread across on-premises servers, Kubernetes, and cloud providers, and exposes industry-standard APIs for LLM, voice, image, and video models.

Within the inference and serving category, GPUStack sits one layer above a single serving engine: instead of running one vLLM process by hand, you let GPUStack pick engines, tune parameters, allocate GPUs, and add operational features like authentication, load balancing, monitoring, and usage metering.

## What it does

- Manages GPU clusters across on-premises servers, Kubernetes, and cloud providers from one server
- Automatically configures inference engines including vLLM, SGLang, and TensorRT-LLM, with support for custom engines
- Pre-tuned modes for low latency or high throughput, plus KV cache systems (LMCache, HiCache) and speculative decoding (EAGLE3, MTP, N-grams)
- Built-in user authentication, access control, load balancing, and automated failure recovery
- Real-time GPU monitoring and token/request metering via integrated Grafana and Prometheus dashboards
- Supports a wide range of accelerators: NVIDIA, AMD, Ascend NPU, Hygon DCU, MThreads, Iluvatar, MetaX, Cambricon MLU, and T-Head PPU

## Getting started

Install the GPUStack server with Docker, log into the UI, then add a GPU cluster and worker nodes. Worker nodes require Linux, an NVIDIA driver, Docker, and the NVIDIA Container Toolkit.

1. Install and start the GPUStack server

   Run the server in Docker. It can run on a CPU-only machine; if you have no dedicated CPU node, install it on the same machine as a GPU worker.

```bash
sudo docker run -d --name gpustack \
    --restart unless-stopped \
    -p 80:80 \
    --volume gpustack-data:/var/lib/gpustack \
    gpustack/gpustack
```

2. Get the admin password and log in

   Retrieve the default admin password, then open http://your_host_ip in a browser and log in with username admin.

```bash
sudo docker exec gpustack cat /var/lib/gpustack/initial_admin_password
```

3. Add a cluster and connect a worker

   In the UI, go to Clusters, click Add Cluster, pick Docker as the provider, and save. The UI gives you a docker run command to register each worker node, similar to the one below.

```bash
sudo docker run -d --name gpustack-worker \
      --restart=unless-stopped \
      --privileged \
      --network=host \
      --volume /var/run/docker.sock:/var/run/docker.sock \
      --volume gpustack-data:/var/lib/gpustack \
      --runtime nvidia \
      gpustack/gpustack \
      --server-url http://your_gpustack_server_url \
      --token your_worker_token \
      --advertise-address 192.168.1.2
```

## When to use it

- Pooling GPUs from several on-prem and cloud machines into one cluster to serve models behind a single API
- Running Model-as-a-Service for internal teams with authentication, access control, and per-user token metering
- Deploying new models on release day by relying on pluggable engines like vLLM, SGLang, or TensorRT-LLM
- Tuning serving for low latency or high throughput using KV cache and speculative decoding options without hand-configuring each engine