Intel Neural Compressor

// Overview

Intel Neural Compressor is an open-source Python library that applies model compression techniques to deep learning models. It supports popular methods such as static and dynamic quantization, SmoothQuant, weight-only quantization, quantization-aware training, and mixed precision, and works across PyTorch, TensorFlow, and JAX.

It is aimed at engineers who want to shrink a trained model and lower its precision so it runs faster and uses less memory at inference time. The library includes advanced quantization paths for large language models and vision-language models such as LLaMA, Qwen, and DeepSeek, through its integration with AutoRound.

Within the quantization and compression space, the toolkit leans toward Intel hardware. It has extensive testing on Intel Gaudi AI Accelerators, Core Ultra, and Xeon Scalable processors, plus Intel data center GPUs, with more limited testing on AMD CPU, ARM CPU, and NVIDIA GPU.

// What it does

• Multiple quantization methods, including static, dynamic, SmoothQuant, weight-only, quantization-aware training, and mixed precision
• Works across PyTorch, TensorFlow, and JAX through framework-specific install packages
• LLM and VLM quantization for models like LLaMA, Qwen, and DeepSeek via AutoRound integration
• Low-precision data types including FP8, NVFP4, and MXFP8 / MXFP4 (several marked experimental)
• Loads pre-quantized Hugging Face models, for example auto-gptq weight-only LLMs, and caches a converted format for later loads
• Tested on Intel Gaudi, Core Ultra, and Xeon hardware, with limited testing on AMD CPU, ARM CPU, and NVIDIA GPU

// Getting started

Install the package that matches your framework backend, then run a short quantization program. The example below performs FP8 quantization and is supported on the Intel Gaudi2 AI Accelerator.

# Framework extension API + PyTorch dependency
pip install neural-compressor-pt
# Framework extension API + TensorFlow dependency
pip install neural-compressor-tf
# Framework extension API + JAX dependency, available since v3.8
# JAX only support build from source installation method
INC_JAX_ONLY=1 pip install .

from neural_compressor.torch.quantization import (
    FP8Config,
    prepare,
    convert,
)

import torch
import torchvision.models as models

model = models.resnet18()
qconfig = FP8Config(fp8_config="E4M3")
model = prepare(model, qconfig)

# Customer defined calibration. Below is a dummy calibration
model(torch.randn(1, 3, 224, 224).to("hpu"))

model = convert(model)

output = model(torch.randn(1, 3, 224, 224).to("hpu")).to("cpu")
print(output.shape)

from neural_compressor.torch.quantization import load

model_name = "TheBloke/Llama-2-7B-GPTQ"
model = load(
    model_name_or_path=model_name,
    format="huggingface",
    device="hpu",
    torch_dtype=torch.bfloat16,
)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Quantize a trained PyTorch or TensorFlow model to lower precision to cut its memory footprint and speed up inference
• Apply weight-only or FP8 quantization to large language models like LLaMA, Qwen, or DeepSeek before serving them
• Run quantization-aware training or SmoothQuant to keep accuracy while reducing precision
• Deploy compressed models on Intel hardware such as Gaudi accelerators or Xeon CPUs

// How Intel Neural Compressor compares

Intel Neural Compressor alongside other open-source quantization & compression tools AI/TLDR tracks, ranked by GitHub stars.