bitsandbytes

// Overview

bitsandbytes is a Python library that brings k-bit quantization to PyTorch. It lets you load and run large language models using 8-bit (LLM.int8()) or 4-bit (QLoRA) weights, which roughly halves or quarters the memory a model needs compared to full 16- or 32-bit precision.

It is aimed at machine-learning engineers and researchers who want to run or fine-tune large models on limited hardware. The library ships drop-in quantized layers (bitsandbytes.nn.Linear8bitLt and bitsandbytes.nn.Linear4bit) and 8-bit optimizers (bitsandbytes.optim), and it is wired into the Hugging Face Transformers, Diffusers, and PEFT stacks so you rarely call it directly.

As a quantization and compression tool, it sits between your model and the accelerator: it stores weights in fewer bits and handles the math needed to keep accuracy close to the original. This is what makes single-GPU inference and QLoRA fine-tuning of large models practical.

// What it does

• LLM.int8() 8-bit inference that runs large models with about half the memory and no measurable quality loss
• QLoRA 4-bit quantization that pairs frozen 4-bit weights with trainable LoRA adapters for memory-efficient fine-tuning
• 8-bit optimizers using block-wise quantization to keep near-32-bit training quality at a fraction of the optimizer memory
• Drop-in quantized layers: bitsandbytes.nn.Linear8bitLt and bitsandbytes.nn.Linear4bit
• Integrated with Hugging Face Transformers, Diffusers, and PEFT for one-flag quantized loading
• Broad accelerator coverage across NVIDIA, AMD, and Intel GPUs plus CPU, with partial Apple Metal support

// Getting started

Most people use bitsandbytes through Hugging Face Transformers, where a single flag loads a model in 8-bit or 4-bit. Install it first, then load a quantized model.

pip install bitsandbytes

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-1.3b",
    load_in_4bit=True,
    device_map="auto",
)

import bitsandbytes as bnb

linear = bnb.nn.Linear8bitLt(768, 768, has_fp16_weights=False)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Run a large language model for inference on a single consumer GPU using 8-bit or 4-bit weights
• Fine-tune a large model with QLoRA when full-precision training would not fit in GPU memory
• Cut optimizer memory during training by switching to bitsandbytes 8-bit optimizers
• Load quantized models in Hugging Face Transformers, Diffusers, or PEFT with a single flag

// How bitsandbytes compares

bitsandbytes alongside other open-source quantization & compression tools AI/TLDR tracks, ranked by GitHub stars.