AWQ

// Overview

AWQ (Activation-aware Weight Quantization) is the reference implementation from MIT Han Lab for shrinking large language models down to 3-bit or 4-bit weights. Instead of treating every weight the same, it looks at which weights matter most for the model's activations and protects them during quantization, so the smaller model keeps most of its accuracy.

It is meant for engineers who need to run LLMs and vision-language models on tighter hardware budgets, from cloud GPUs down to edge devices like the NVIDIA Jetson Orin. The repo ships an AWQ search step, a pre-computed model zoo for families such as Llama-1/2/3, OPT, CodeLlama, StarCoder, Vicuna, and VILA, plus a memory-efficient 4-bit Linear layer and CUDA kernels for fast inference.

As a quantization and compression tool, AWQ sits ahead of the serving stack: you quantize a model once, then load the result. The method is also integrated into wider tooling such as Hugging Face Transformers, vLLM, TensorRT-LLM, and LMDeploy, and the companion TinyChat project uses it for on-device LLM/VLM chat.

// What it does

• Activation-aware INT3/INT4 weight quantization that keeps accuracy close to the full-precision model
• AWQ search to find quantization scales, plus a real backend that dumps deployable quantized weights
• Pre-computed AWQ model zoo for Llama-1/2/3, OPT, CodeLlama, StarCoder, Vicuna, VILA, and LLaVA
• Memory-efficient 4-bit Linear layer in PyTorch with CUDA kernels for context and decoding stages
• Support for instruction-tuned and multi-modal models (e.g. Vicuna, VILA) including video understanding
• TinyChat companion for on-device LLM/VLM inference on edge GPUs, reporting up to ~2.7-2.9x speedups over FP16

// Getting started

AWQ runs as a Python package with a CUDA kernel you build from source. Clone the repo, set up the environment, then run the AWQ search and quantization steps.

git clone https://github.com/mit-han-lab/llm-awq
cd llm-awq
conda create -n awq python=3.10 -y
conda activate awq
pip install --upgrade pip
pip install -e .
cd awq/kernels
python setup.py install

python -m awq.entry --model_path /PATH/TO/LLAMA3/llama3-8b \
    --w_bit 4 --q_group_size 128 \
    --run_awq --dump_awq awq_cache/llama3-8b-w4-g128.pt

mkdir quant_cache
python -m awq.entry --model_path /PATH/TO/LLAMA3/llama3-8b \
    --w_bit 4 --q_group_size 128 \
    --load_awq awq_cache/llama3-8b-w4-g128.pt \
    --q_backend real --dump_quant quant_cache/llama3-8b-w4-g128-awq.pt

python -m awq.entry --model_path /PATH/TO/LLAMA3/llama3-8b \
    --tasks wikitext --w_bit 4 --q_group_size 128 \
    --load_quant quant_cache/llama3-8b-w4-g128-awq.pt

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Compressing a Llama-3 or other LLM to 4-bit so it fits in less GPU memory while keeping accuracy
• Running LLM or vision-language chatbots on edge devices like Jetson Orin with TinyChat
• Producing AWQ-quantized checkpoints to serve through vLLM, TensorRT-LLM, or LMDeploy
• Cutting inference cost and latency for instruction-tuned and multi-modal models

// How AWQ compares

AWQ alongside other open-source quantization & compression tools AI/TLDR tracks, ranked by GitHub stars.