LLM Compressor

// Overview

LLM Compressor (`llmcompressor`) is a Python library for shrinking large language models so they run faster and use less memory when served. It applies quantization and sparsity methods to a model's weights, activations, KV cache, and attention, then saves the result in the `compressed-tensors` format.

It is built for engineers who deploy models with vLLM. Models load directly from Hugging Face repositories, get compressed with a chosen scheme, and are saved back out ready for vLLM to serve. It also supports DDP and disk offloading so very large models can be compressed.

Within the quantization and compression category, it sits between your trained model and your serving stack: you pick a precision (such as FP8, INT8, INT4, or NVFP4) and an algorithm (such as GPTQ, AWQ, or SmoothQuant), and it produces a smaller checkpoint that vLLM can run.

// What it does

• Quantization for weights, activations, KV cache, and attention, including W8A8 (int8/fp8), W4A16, and microscale formats like NVFP4, MXFP4, and MXFP8
• A range of algorithms: simple PTQ, GPTQ, AWQ, SmoothQuant, AutoRound, and rotation-based methods (SpinQuant, QuIP)
• Loads Hugging Face models and repositories directly and saves to the compressed-tensors format that vLLM can serve
• DDP and disk offloading support for compressing very large models
• Works across model types, including MoE LLMs and vision-, audio-, and language multimodal models

// Getting started

Install the library, then run a one-shot quantization pass on a Hugging Face model and save the compressed result for vLLM.

pip install llmcompressor

from compressed_tensors.offload import dispatch_model
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "Qwen/Qwen3-30B-A3B"

# Load model.
model = AutoModelForCausalLM.from_pretrained(MODEL_ID)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Configure the quantization algorithm and scheme.
recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_BLOCK",
    ignore=["lm_head", "re:.*mlp.gate$"],
)

# Apply quantization.
oneshot(model=model, recipe=recipe)

# Save to disk in compressed-tensors format.
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-BLOCK"
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reduce the memory footprint and serving cost of a model before deploying it with vLLM
• Produce FP8 or INT4 checkpoints from a Hugging Face model to fit on smaller or fewer GPUs
• Compress very large models using DDP and disk offloading when they don't fit in memory
• Quantize MoE or multimodal (vision/audio) models for production inference

// How LLM Compressor compares

LLM Compressor alongside other open-source quantization & compression tools AI/TLDR tracks, ranked by GitHub stars.