# ExLlamaV3

> Run quantized local LLMs on consumer GPUs with the EXL3 format

- Category: Models & Inference / Local Runtimes
- Language: Python
- Repository: https://github.com/turboderp-org/exllamav3
- Canonical page: https://ai-tldr.dev/tools/exllamav3/

## In short

ExLlamaV3 is an inference library for running local large language models on modern consumer GPUs. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Python. To get started: install from a prebuilt wheel (recommended).

## Overview

ExLlamaV3 is an inference library for running local large language models on modern consumer GPUs. It is the next version of the ExLlama project and introduces the EXL3 quantization format, based on QTIP, which lets you fit larger models into limited VRAM while keeping inference fast.

It is built for people who want to run models on their own hardware rather than calling a hosted API. You can quantize a Hugging Face model to EXL3 and serve it through TabbyAPI, the recommended backend that exposes an OpenAI-compatible endpoint. It also plugs into HF Transformers as a backend.

As a local runtime, it sits between your downloaded model weights and the application that talks to them. It supports tensor-parallel and expert-parallel inference, continuous dynamic batching, speculative decoding, cache quantization, multimodal models, and LoRA adapters across a wide range of model architectures.

## What it does

- EXL3 quantization format based on QTIP for fitting larger models into limited GPU memory
- Tensor-parallel and expert-parallel inference across consumer multi-GPU setups
- Continuous, dynamic batching for serving multiple requests at once
- OpenAI-compatible server through the TabbyAPI backend, plus a HF Transformers plugin
- Speculative decoding, 2-8 bit cache quantization, multimodal support, and LoRA adapters
- Broad model architecture support including Llama, Mistral, Qwen, Gemma, GLM, and Command-R

## Getting started

Make sure you have a CUDA 12.4 or later build of PyTorch installed first, since the Torch dependency is not handled automatically by pip. Then install ExLlamaV3 and convert a model to EXL3.

1. Install from a prebuilt wheel (recommended)

   Pick a wheel matching your CUDA, Torch, and Python version from the releases page, then install it directly with pip. This is the simplest path if you are unsure which method to use.

```bash
pip install https://github.com/turboderp-org/exllamav3/releases/download/v0.0.6/exllamav3-0.0.6+cu128.torch2.8.0-cp313-cp313-linux_x86_64.whl
```

2. Or install from PyPI

   The PyPI package has no prebuilt extension, so it needs the CUDA toolkit and build prerequisites (VS Build Tools on Windows, gcc on Linux, python-dev headers).

```bash
pip install exllamav3
```

3. Convert a model to EXL3

   Use the conversion script to quantize a Hugging Face model. The working directory holds checkpoints and quantized tensors and needs room for a full copy of the output model.

```bash
python convert.py -i <input_dir> -o <output_dir> -w <working_dir> -b <bitrate>
```

4. Serve with TabbyAPI

   TabbyAPI is the recommended backend and provides an OpenAI-compatible API, with a startup script that installs prerequisites and manages inference. Point it at your converted EXL3 model to start serving.

## When to use it

- Run a large open-weight model on a single consumer GPU by quantizing it to EXL3 to save VRAM
- Stand up a private, OpenAI-compatible LLM endpoint on your own hardware via TabbyAPI
- Spread inference across multiple consumer GPUs using tensor-parallel or expert-parallel modes
- Use it as a faster local backend behind HF Transformers for experimentation