Overview
Optimum Quanto is a PyTorch quantization backend for Hugging Face Optimum. It converts a float model into a quantized one, reducing the precision of weights and activations to formats like int2, int4, int8, and float8. The main goal is to cut the device memory a model needs while keeping accuracy close to the full-precision version.
It is aimed at PyTorch and Hugging Face users who want to run large models, such as LLMs and diffusion models, on smaller or cheaper hardware. It works in eager mode (including non-traceable models), can place quantized models on CUDA or MPS devices, and saves models with standard PyTorch weight_only or safetensors serialization.
As a quantization and compression tool, it fits between your float model and inference. Note that the project is in maintenance mode: the README points to bitsandbytes or torchAO for production-ready or actively developed quantization.
What it does
- Supports int2, int4, int8, and float8 weights, plus int8 and float8 activations
- Works in eager mode, including with non-traceable models, and runs on CUDA and MPS devices
- Helper classes (QuantizedModelForCausalLM, QuantizedPixArtTransformer2DModel) to quantize, save, and reload Hugging Face LLM and diffusers models
- Accelerated matrix multiplications on CUDA (int8-int8, fp16-int4, bf16-int8, bf16-int4)
- Low-level API with a dynamic-to-static workflow: quantize, calibrate, optionally tune (QAT), then freeze
- Serialization compatible with PyTorch weight_only and Hugging Face safetensors
Getting started
Optimum Quanto ships as a pip package. Install it, then quantize a Hugging Face model with one of the helper classes.
Install
Install the package from PyPI.
pip install optimum-quantoQuantize an LLM
Load a model and quantize its weights, for example to int4. You can exclude layers such as lm_head.
from transformers import AutoModelForCausalLM
from optimum.quanto import QuantizedModelForCausalLM, qint4
model = AutoModelForCausalLM.from_pretrained('meta-llama/Meta-Llama-3-8B')
qmodel = QuantizedModelForCausalLM.quantize(model, weights=qint4, exclude='lm_head')Save and reload
Save the quantized model, then load it back later with from_pretrained.
qmodel.save_pretrained('./Llama-3-8B-quantized')
from optimum.quanto import QuantizedModelForCausalLM
qmodel = QuantizedModelForCausalLM.from_pretrained('Llama-3-8B-quantized')Quantize vanilla PyTorch models
For plain PyTorch models, use the low-level API: quantize, then freeze to replace float weights with quantized integer weights.
from optimum.quanto import quantize, qint8, freeze
quantize(model, weights=qint8, activations=qint8)
freeze(model)Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Run a large LLM on a single GPU by quantizing its weights to int4 or int8 to reduce memory use
- Quantize the transformer submodel of a diffusers pipeline (e.g. PixArt) to float8 and reload it into a pipeline
- Apply int8 or float8 quantization to a custom PyTorch model and recover accuracy with calibration or quantization-aware tuning
- Save quantized models as safetensors for portable, weight_only-compatible reloading
How Optimum Quanto compares
Optimum Quanto alongside other open-source quantization & compression tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| BitNet | ★ 39.6k | Microsoft's inference framework for running extremely low-bit (1.58-bit) quantized LLMs efficiently on CPUs. |
| bitsandbytes | ★ 8.3k | A library that adds 8-bit and 4-bit quantization to PyTorch models, widely used for loading and fine-tuning LLMs in low memory. |
| AWQ | ★ 3.6k | The reference implementation of Activation-aware Weight Quantization for compressing LLMs to 4-bit with little accuracy loss. |
| LLM Compressor | ★ 3.5k | A library for applying quantization and sparsity methods to LLMs to produce compressed models that run faster in vLLM. |
| torchao | ★ 2.9k | PyTorch's native library for quantizing and applying low-bit data types to models for faster training and inference. |
| Intel Neural Compressor | ★ 2.7k | An Intel toolkit for compressing models through quantization, pruning, and distillation across multiple deep-learning frameworks. |
| GPTQModel | ★ 1.2k | A model quantization toolkit, successor to AutoGPTQ, that compresses LLMs with GPTQ, AWQ, and other methods across many chips. |
| Optimum Quanto | ★ 1k | A PyTorch quantization backend for Optimum that shrinks model weights and activations to int and float8 |