Multiple institutions · 2026-05-01 · notable
AGoQ — 4-Bit Activation + 8-Bit Gradient Quantization Cuts Distributed LLM Training Memory by 52%
AGoQ combines layer-aware 4-bit activation quantization and precision-preserving 8-bit gradient All-Reduce to cut LLM training memory by up to 52% and speed up training by 1.34x on 64-GPU clusters, beating Megatron-LM, COAT, and DeepSpeed.
AGoQ achieves near 4-bit activation storage with 8-bit gradient communication — halving memory without hurting convergence.
Key specs
| Memory reduction | 52% |
|---|---|
| Training speedup | 1.34x |
| Gpus tested | 64 |
| Model sizes tested | 8B-32B |
What is it?
AGoQ (Activation and Gradient Quantization) is a memory reduction framework for distributed LLM pretraining submitted to arXiv on May 1, 2026. It targets the two dominant memory consumers during training: stored activations needed for backpropagation and gradient tensors sent across accelerators during All-Reduce.
How does it work?
A layer-aware activation quantization algorithm assigns different bit-widths based on each layer's type and pipeline stage, achieving near 4-bit storage for activations. For gradients, it employs 8-bit storage combined with a precision-preserving 8-bit All-Reduce communication protocol that avoids the compounding errors typical of low-precision collective operations.
Why does it matter?
Training 8B–32B models with ZeRO or pipeline parallelism still requires significant GPU memory for activations and gradient buffering. AGoQ provides a practical toolkit that outperforms Megatron-LM, DeepSpeed, and COAT in both memory and speed while maintaining convergence quality — reducing barriers to training larger models on the same hardware.