# MACaQ

> One 4-bit quantization recipe for text, image, video and audio models

- Category: Models & Inference / Quantization & Compression
- Language: C++
- Licence: Apache-2.0
- Repository: https://github.com/huawei-csl/MACaQ
- Website: https://huawei-csl.com/blog/macaq/
- Canonical page: https://ai-tldr.dev/tools/macaq/
- Updated: 2026-11-10

## In short

MACaQ (Model Agnostic Calibrated Quantization) is an open-source quantization toolkit from Huawei Computing Systems Lab. It sits in the Quantization & Compression category of the Models & Inference tools and is open-source, written in C++ and licensed Apache-2.0. To get started: create the vLLM environment.

## Overview

MACaQ (Model Agnostic Calibrated Quantization) is an open-source quantization toolkit from Huawei Computing Systems Lab. It compresses a model's weights, and optionally its activations, to 4 bits or less so the model needs less memory and runs faster, while keeping the output close to the original. It works with text, image, video and audio models and produces files for llama.cpp (GGUF), vLLM, Diffusers and plain PyTorch.

The method treats each linear layer on its own. For every layer MACaQ applies a per-channel scale and a block-Hadamard rotation to the activations at inference time and folds the inverse transforms into the weights, so nothing has to be absorbed into neighbouring normalization or residual layers. Because no transform crosses a layer boundary, the same recipe applies to new architectures without model-specific engineering, and layers can be quantized one at a time. A second step, Flip-Refinement, nudges individual FP4 codes with a Hessian-aware coordinate descent to beat plain round-to-nearest.

The authors report that MACaQ quantizes a 122B-parameter model in 64 minutes with 32.5 GB of device memory, and that it matches the best Llama-3-8B perplexity of MR-GPTQ and DuQuant++ with 11 times less quantization time. On Qwen3.6-35B-A3B its refined Q4_K_XL GGUF has a lower KL divergence to FP16 (0.013367) than Unsloth's UD-Q4_K_XL file of the same 22.36 GB size (0.015731). The code is Apache-2.0 and the method is described in the arXiv paper 2611.10649.

## What it does

- Weight-and-activation quantization to 4-bit MXFP4 (W4A4), plus 2-, 3- and 4-bit GGUF weight formats for llama.cpp
- Layer-local transforms: a per-channel scale and block-Hadamard rotation per linear layer, with no cross-layer dependencies
- Flip-Refinement, a Hessian-aware coordinate-descent pass that improves FP4 codes over round-to-nearest
- One recipe across modalities: LLMs and MoE models, FLUX.2 [klein] 4B images, Wan2.2 video and Qwen3-TTS speech
- Backends for llama.cpp, vLLM, Diffusers and PyTorch; llama.cpp also runs on CPU
- W4A4 dense and grouped matmul kernels for Huawei Ascend, submitted upstream to CANN

## Getting started

MACaQ runs on Linux with Python 3.12 and uses a separate virtual environment per backend. The project blog walks through vLLM and llama.cpp; the vLLM path is shown here, taken from that guide.

1. Create the vLLM environment

   Clone the repository, then create a Python 3.12 environment for the vLLM backend and install MACaQ into it.

```bash
python3.12 -m venv .venv-vllm
.venv-vllm/bin/pip install -r requirements/vllm-qwen35.txt
.venv-vllm/bin/pip install -r requirements/build.txt
.venv-vllm/bin/pip install --no-build-isolation --no-deps -e .
```

2. Quantize a model

   Point MACaQ at a Hugging Face model and a file of representative UTF-8 calibration text. It writes a W4A4 checkpoint to the output folder.

```bash
.venv-vllm/bin/python vllm/cli.py quantize \
  --model Qwen/Qwen3.8-27B \
  --calibration data/calibration.txt \
  --output models/qwen38-27b-macaq --gpu 0
```

3. Serve the quantized checkpoint

   Serve the saved checkpoint without recalibrating. It exposes an OpenAI-style chat endpoint at http://127.0.0.1:8000/v1/chat/completions.

```bash
.venv-vllm/bin/python vllm/cli.py serve \
  --model models/qwen38-27b-macaq --gpu 0
```

4. Or refine an existing GGUF for llama.cpp

   In the llama.cpp environment, MACaQ-R refines an existing 4-bit GGUF against the FP16 teacher. The result runs with the normal llama CLI or server.

```bash
.venv-llamacpp/bin/python llamacpp/macaq.py refine \
  --model models/model-q4.gguf --hf models/original \
  --teacher models/original-f16.gguf --calibration data/calibration.txt \
  --output models/model-macaq-r.gguf --work-dir work/refinement

llama serve -m models/model-macaq-r.gguf
```

## When to use it

- Make a smaller, higher-quality 4-bit GGUF of an open model to run locally with llama.cpp
- Serve an LLM in vLLM with 4-bit weights and activations for faster prefill and decode on the same GPU
- Quantize image, video or speech models such as FLUX.2 [klein] 4B, Wan2.2 or Qwen3-TTS with the same recipe used for LLMs
- Quantize a brand-new architecture without writing model-specific transform code

## Version history

- 2026-11-10: Huawei Computing Systems Lab open-sourced MACaQ under Apache-2.0, with llama.cpp, vLLM, Diffusers and PyTorch backends and the arXiv paper describing the method.