GPTQModel

// Overview

GPTQModel is a Python toolkit for quantizing large language models. It is the successor to AutoGPTQ and lets you compress a model's weights to lower bit-widths, such as 4-bit, so the model takes less memory and runs faster while keeping output quality close to the original.

It is aimed at ML engineers and practitioners who want to deploy LLMs on limited hardware. You load a model with a quantization config, calibrate it on a small dataset, and save a compressed checkpoint you can serve later.

Within the quantization and compression space, it bundles several methods (GPTQ, AWQ, GGUF, FP8 and others) and runs across NVIDIA GPUs, Intel CPU/XPU and Huawei Ascend NPU, and it integrates with the Hugging Face Transformers, PEFT and Optimum stack.

// What it does

• Supports several quantization methods, including GPTQ, AWQ, GGUF, FP8, EXL3, QQQ and ParoQuant
• Runs across a range of hardware: NVIDIA GPUs, Intel CPU and XPU, and Huawei Ascend NPU through native torch kernels
• JIT-compiled CUDA kernels that build only what you use, shrinking the wheel and skipping unused code paths
• Wide model coverage across many recent LLM and MoE families, with handling for Mixture-of-Experts routing during quantization
• Works alongside the Hugging Face stack, with support for Transformers, PEFT and Optimum for both GPTQ and AWQ
• Optional extras for serving backends such as vLLM, SGLang and BitBLAS

// Getting started

Install the package from PyPI, then load a model with a quantization config, calibrate it on a small dataset, and save the quantized weights.

pip install -v gptqmodel

from datasets import load_dataset
from gptqmodel import GPTQConfig, GPTQModel

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train"
).select(range(1024))["text"]

quant_config = GPTQConfig(bits=4, group_size=128)
model = GPTQModel.load(model_id, quant_config)
model.quantize(calibration_dataset, batch_size=1)
model.save(quant_path)

model = GPTQModel.load("ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v2.5")
result = model.generate("Uncovering deep insights begins with")[0]
print(model.tokenizer.decode(result))

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Shrink a large language model to 4-bit so it fits on a smaller GPU or on multiple consumer cards
• Prepare quantized checkpoints to serve with backends like vLLM or SGLang for cheaper, faster inference
• Compress Mixture-of-Experts models using the routing and fail-safe controls built for uneven expert quantization
• Quantize models for non-NVIDIA hardware such as Intel CPU/XPU or Huawei Ascend NPU

// How GPTQModel compares

GPTQModel alongside other open-source quantization & compression tools AI/TLDR tracks, ranked by GitHub stars.