MACaQ

// Overview

MACaQ (Model Agnostic Calibrated Quantization) is an open-source quantization toolkit from Huawei Computing Systems Lab. It compresses a model's weights, and optionally its activations, to 4 bits or less so the model needs less memory and runs faster, while keeping the output close to the original. It works with text, image, video and audio models and produces files for llama.cpp (GGUF), vLLM, Diffusers and plain PyTorch.

The method treats each linear layer on its own. For every layer MACaQ applies a per-channel scale and a block-Hadamard rotation to the activations at inference time and folds the inverse transforms into the weights, so nothing has to be absorbed into neighbouring normalization or residual layers. Because no transform crosses a layer boundary, the same recipe applies to new architectures without model-specific engineering, and layers can be quantized one at a time. A second step, Flip-Refinement, nudges individual FP4 codes with a Hessian-aware coordinate descent to beat plain round-to-nearest.

The authors report that MACaQ quantizes a 122B-parameter model in 64 minutes with 32.5 GB of device memory, and that it matches the best Llama-3-8B perplexity of MR-GPTQ and DuQuant++ with 11 times less quantization time. On Qwen3.6-35B-A3B its refined Q4_K_XL GGUF has a lower KL divergence to FP16 (0.013367) than Unsloth's UD-Q4_K_XL file of the same 22.36 GB size (0.015731). The code is Apache-2.0 and the method is described in the arXiv paper 2611.10649.

// What it does

• Weight-and-activation quantization to 4-bit MXFP4 (W4A4), plus 2-, 3- and 4-bit GGUF weight formats for llama.cpp
• Layer-local transforms: a per-channel scale and block-Hadamard rotation per linear layer, with no cross-layer dependencies
• Flip-Refinement, a Hessian-aware coordinate-descent pass that improves FP4 codes over round-to-nearest

• One recipe across modalities: LLMs and MoE models, FLUX.2 [klein] 4B images, Wan2.2 video and Qwen3-TTS speech
• Backends for llama.cpp, vLLM, Diffusers and PyTorch; llama.cpp also runs on CPU
• W4A4 dense and grouped matmul kernels for Huawei Ascend, submitted upstream to CANN

// Getting started

MACaQ runs on Linux with Python 3.12 and uses a separate virtual environment per backend. The project blog walks through vLLM and llama.cpp; the vLLM path is shown here, taken from that guide.

python3.12 -m venv .venv-vllm
.venv-vllm/bin/pip install -r requirements/vllm-qwen35.txt
.venv-vllm/bin/pip install -r requirements/build.txt
.venv-vllm/bin/pip install --no-build-isolation --no-deps -e .

.venv-vllm/bin/python vllm/cli.py quantize \
  --model Qwen/Qwen3.8-27B \
  --calibration data/calibration.txt \
  --output models/qwen38-27b-macaq --gpu 0

.venv-vllm/bin/python vllm/cli.py serve \
  --model models/qwen38-27b-macaq --gpu 0

.venv-llamacpp/bin/python llamacpp/macaq.py refine \
  --model models/model-q4.gguf --hf models/original \
  --teacher models/original-f16.gguf --calibration data/calibration.txt \
  --output models/model-macaq-r.gguf --work-dir work/refinement

llama serve -m models/model-macaq-r.gguf

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Make a smaller, higher-quality 4-bit GGUF of an open model to run locally with llama.cpp
• Serve an LLM in vLLM with 4-bit weights and activations for faster prefill and decode on the same GPU
• Quantize image, video or speech models such as FLUX.2 [klein] 4B, Wan2.2 or Qwen3-TTS with the same recipe used for LLMs
• Quantize a brand-new architecture without writing model-specific transform code

// How MACaQ compares

MACaQ alongside other open-source quantization & compression tools AI/TLDR tracks, ranked by GitHub stars.