Overview
Lucebox is an LLM inference server aimed at the hardware most people actually own: a single consumer card, or a box with a mismatched pair of them. Rather than assuming a datacentre GPU and scaling down, it starts from specific cards — RTX 3090 and 4090 on the NVIDIA side, Radeon R9700 and Strix Halo on the AMD side — and hand-tunes kernels for each, which is why its build flags name GPU architectures directly.
The performance story is speculative execution applied to both halves of inference: speculative prefill as well as speculative decoding, layered on top of paged attention and continuous batching so throughput does not collapse under concurrent requests. Several optimisation paths ship under their own names — DFlash2, PFlash, KVFlash, Spark and a megakernel implementation — and which one applies depends on the model and card.
Heterogeneous setups are a first-class case rather than an afterthought: CUDA and ROCm backends can be built from the same tree, so an NVIDIA and an AMD card in one machine are both usable. The project is written in C++ with Python tooling around it, Apache-2.0 licensed, and built from source with CMake and Ninja.
What it does
- Speculative prefill and speculative decoding, not just speculative decode
- Hand-written kernels targeted at specific consumer GPUs (RTX 3090/4090, Radeon R9700, Strix Halo)
- Heterogeneous NVIDIA CUDA + AMD ROCm support from one source tree
- Paged attention with continuous batching for concurrent requests
- Multiple optimisation paths — DFlash2, PFlash, KVFlash, Spark and a megakernel implementation
- Apache-2.0, built from source with CMake and Ninja
Getting started
Lucebox is built from source. The build selects a GPU backend and architecture explicitly — the snippet below is the ROCm 7.2+ / RDNA4 path from the README; the CUDA path uses the equivalent NVIDIA flags.
Clone with submodules
The kernels live in submodules, so a plain clone is not enough.
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd luceboxConfigure the build for your GPU
Set the backend (hip here) and the architecture of the card you are targeting; gfx1201 is RDNA4.
cmake -S server -B server/build-hip -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
-DDFLASH27B_GPU_BACKEND=hip \
-DDFLASH27B_HIP_ARCHITECTURES=gfx1201Build the server
Compiles the inference server binary against the backend you configured.
cmake --build server/build-hip --target dflash_server -j"$(nproc)"Read the setup and benchmark guides
The repository carries per-card setup notes, API documentation, an architecture write-up and benchmark methodology — worth reading before tuning, since the right optimisation path depends on the model and the card.
Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Get usable tokens per second out of a single RTX 3090 or 4090 instead of renting a datacentre GPU
- Serve a model from a machine with an NVIDIA and an AMD card without running two stacks
- Squeeze a local coding or chat assistant into consumer hardware where speculative decoding pays off most
- Compare speculative prefill/decode against a general-purpose server like vLLM or SGLang on your own card
How Lucebox compares
Lucebox alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Transformers | ★ 165k | Hugging Face Transformers is a Python framework that defines and runs state-of-the-art pretrained models for text, vision, audio, and multimodal tasks, for both inference and training. |
| vLLM | ★ 91.4k | A high-throughput LLM serving engine that uses PagedAttention and continuous batching to serve many requests at once. |
| SGLang | ★ 35.8k | A serving framework for LLMs and multimodal models that boosts throughput by reusing shared prompt prefixes across requests. |
| Modular Platform | ★ 29.7k | Modular's AI platform: the MAX inference framework with an OpenAI-compatible serving endpoint and GPU kernel library, plus the Mojo language and compiler. |
| TensorRT-LLM | ★ 14.6k | NVIDIA's library that compiles LLMs into optimized engines for the fastest inference on its data-center GPUs. |
| OpenLLM | ★ 12.5k | A tool to run any open-source LLM as an OpenAI-compatible API endpoint locally or in the cloud. |
| LMCache | ★ 11.7k | A KV-cache layer that stores and shares cached attention state across engines and requests to cut repeated computation. |
| Lucebox | ★ 2.8k | Speculative-decoding inference tuned for one consumer GPU at a time |