# ExLlamaV2

> Run quantized EXL2 LLMs locally on consumer NVIDIA GPUs

- Category: Models & Inference / Local Runtimes
- Language: Python
- Repository: https://github.com/turboderp-org/exllamav2
- Canonical page: https://ai-tldr.dev/tools/exllamav2/

## In short

ExLlamaV2 is an inference library for running local large language models on modern consumer NVIDIA GPUs. It focuses on the EXL2 quantization format (and also supports GPTQ), letting you fit larger models into limited VRAM and generate tokens quickly on a single card. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Python.

## Overview

ExLlamaV2 is an inference library for running local large language models on modern consumer NVIDIA GPUs. It focuses on the EXL2 quantization format (and also supports GPTQ), letting you fit larger models into limited VRAM and generate tokens quickly on a single card.

It is aimed at developers and self-hosters who want to run models on their own hardware instead of a hosted API. You can call it directly as a Python library, or pair it with a server like TabbyAPI to expose an OpenAI-compatible HTTP endpoint for frontends such as SillyTavern, ExUI, or text-generation-webui.

As a local runtime, it sits between your quantized model files and your application. Newer versions add a dynamic generator with batching, smart prompt caching, K/V cache deduplication, and paged attention via Flash Attention. Note that the project is now archived, with development continuing on ExLlamaV3.

## What it does

- Runs quantized LLMs in the EXL2 format, plus GPTQ models, on consumer NVIDIA GPUs
- Dynamic generator with batching, smart prompt caching, and K/V cache deduplication
- Paged attention via Flash Attention 2.5.7+
- Single, batched, and asyncio-streamed generation through one simplified API
- Q4 K/V cache mode to reduce memory use during inference
- Multi-GPU inference with automatic GPU splitting (--gpu_split auto)

## Getting started

Install from source with the CUDA Toolkit and a matching PyTorch build, then run a quick inference test against a local model. Prebuilt wheels are also available if you prefer not to compile the extension.

1. Install from source

   Clone the repo and install. You need the CUDA Toolkit, a compiler (gcc on Linux or Visual Studio Build Tools on Windows), and a matching PyTorch version. This compiles the exllamav2_ext extension.

```bash
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install .
```

2. Run a test inference

   Point the test script at a local EXL2 or GPTQ model. Add --gpu_split auto for multi-GPU setups.

```bash
python test_inference.py -m <path_to_model> -p "Once upon a time,"
```

3. Try the console chatbot

   An example chat script is included. The -mode flag picks the prompt format; run with -modes to list all options.

```bash
python examples/chat.py -m <path_to_model> -mode llama -gs auto
```

4. Generate from Python

   Once a model and generator are set up, the dynamic generator exposes a simple generate call for single or batched prompts.

```python
output = generator.generate(prompt = "Hello, my name is", max_new_tokens = 200)
```

## When to use it

- Self-hosting a quantized LLM on a single consumer GPU instead of using a hosted API
- Serving an OpenAI-compatible local endpoint via TabbyAPI for chat frontends like SillyTavern or ExUI
- Fitting larger models into limited VRAM by running them in the EXL2 format
- Batched or streamed text generation in a Python application or backend service