█

AI/TLDR

BitNet

Run 1.58-bit quantized LLMs fast on your CPU

github.com/microsoft/BitNet★ 39.6k

Overview

BitNet (bitnet.cpp) is Microsoft's official inference framework for 1-bit large language models, such as BitNet b1.58. Instead of the usual 16-bit weights, these models store each weight in about 1.58 bits (the ternary values -1, 0, 1), so the model files and memory use shrink a lot.

The framework ships a set of optimized kernels built on top of llama.cpp that run these ternary models on CPU and GPU. On CPUs it reports speedups of roughly 1.37x to 6.17x over a baseline and large drops in energy use, and it can even run a 100B BitNet b1.58 model on a single CPU at reading speed (5-7 tokens per second).

It is aimed at developers who want to run quantized LLMs locally without a dedicated accelerator, and at researchers exploring 1-bit model inference. Within the quantization and compression space, BitNet focuses specifically on ternary (1.58-bit) models rather than general low-bit formats.

What it does

  • Official inference framework for 1-bit / 1.58-bit ternary LLMs like BitNet b1.58
  • Optimized kernels (I2_S, TL1, TL2) for both x86 and ARM CPUs
  • Reported CPU speedups up to ~6x and large energy-use reductions versus baseline inference
  • Can run a 100B BitNet b1.58 model on a single CPU at 5-7 tokens per second
  • GPU inference kernel available in addition to CPU support
  • Built on the llama.cpp framework, with ready-to-use models on Hugging Face

Getting started

Build bitnet.cpp from source, download a ternary model, then run inference in chat mode. These commands come from the project's README.

Clone the repository

Clone with submodules, since bitnet.cpp pulls in dependencies like llama.cpp.

bashbash
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

Set up the environment

Create a Python environment and install the project's requirements.

bashbash
conda create -n bitnet-cpp python=3.9
conda activate bitnet-cpp
pip install -r requirements.txt

Download a model and build kernels

Download the official 2B model in GGUF format, then run setup_env.py to build the kernels for the i2_s quantization type.

bashbash
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

Run inference

Point run_inference.py at the GGUF model file. Use -p for the prompt and -cnv to enable conversation mode.

bashbash
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Run a quantized LLM locally on a laptop or server CPU without a dedicated GPU
  • Cut memory footprint and energy use for on-device or edge inference
  • Experiment with and benchmark 1-bit / 1.58-bit ternary models from Hugging Face
  • Serve larger BitNet models on commodity hardware where full-precision inference would not fit

How BitNet compares

BitNet alongside other open-source quantization & compression tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
BitNet★ 39.6kRun 1.58-bit quantized LLMs fast on your CPU
bitsandbytes★ 8.3kA library that adds 8-bit and 4-bit quantization to PyTorch models, widely used for loading and fine-tuning LLMs in low memory.
AWQ★ 3.6kThe reference implementation of Activation-aware Weight Quantization for compressing LLMs to 4-bit with little accuracy loss.
LLM Compressor★ 3.5kA library for applying quantization and sparsity methods to LLMs to produce compressed models that run faster in vLLM.
torchao★ 2.9kPyTorch's native library for quantizing and applying low-bit data types to models for faster training and inference.
Intel Neural Compressor★ 2.7kAn Intel toolkit for compressing models through quantization, pruning, and distillation across multiple deep-learning frameworks.
GPTQModel★ 1.2kA model quantization toolkit, successor to AutoGPTQ, that compresses LLMs with GPTQ, AWQ, and other methods across many chips.
Optimum Quanto★ 1kA Hugging Face PyTorch toolkit for quantizing model weights and activations to formats like int8, int4, and float8.