ExLlamaV2

// Overview

ExLlamaV2 is an inference library for running local large language models on modern consumer NVIDIA GPUs. It focuses on the EXL2 quantization format (and also supports GPTQ), letting you fit larger models into limited VRAM and generate tokens quickly on a single card.

It is aimed at developers and self-hosters who want to run models on their own hardware instead of a hosted API. You can call it directly as a Python library, or pair it with a server like TabbyAPI to expose an OpenAI-compatible HTTP endpoint for frontends such as SillyTavern, ExUI, or text-generation-webui.

As a local runtime, it sits between your quantized model files and your application. Newer versions add a dynamic generator with batching, smart prompt caching, K/V cache deduplication, and paged attention via Flash Attention. Note that the project is now archived, with development continuing on ExLlamaV3.

// What it does

• Runs quantized LLMs in the EXL2 format, plus GPTQ models, on consumer NVIDIA GPUs
• Dynamic generator with batching, smart prompt caching, and K/V cache deduplication
• Paged attention via Flash Attention 2.5.7+
• Single, batched, and asyncio-streamed generation through one simplified API
• Q4 K/V cache mode to reduce memory use during inference
• Multi-GPU inference with automatic GPU splitting (--gpu_split auto)

// Getting started

Install from source with the CUDA Toolkit and a matching PyTorch build, then run a quick inference test against a local model. Prebuilt wheels are also available if you prefer not to compile the extension.

git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install .

python test_inference.py -m <path_to_model> -p "Once upon a time,"

python examples/chat.py -m <path_to_model> -mode llama -gs auto

output = generator.generate(prompt = "Hello, my name is", max_new_tokens = 200)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Self-hosting a quantized LLM on a single consumer GPU instead of using a hosted API
• Serving an OpenAI-compatible local endpoint via TabbyAPI for chat frontends like SillyTavern or ExUI
• Fitting larger models into limited VRAM by running them in the EXL2 format
• Batched or streamed text generation in a Python application or backend service

// How ExLlamaV2 compares

ExLlamaV2 alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.