AirLLM

// Overview

AirLLM is an open-source Python library that shrinks the GPU memory an LLM needs at inference time. Instead of holding the whole model in VRAM, it splits the checkpoint into per-layer shards on disk and loads only the layer it is currently computing. Because the VRAM requirement then tracks the size of a single layer rather than the whole model, the project reports running a 70B model on one 4GB card, Llama 3.1 405B on about 8GB, and DeepSeek-V3 (671B) on about 12GB. For sparse Mixture-of-Experts models it streams one expert at a time, which is how it reports fitting Kimi K3 under 4GB.

The trade is memory for time: every layer is read from disk on each forward pass, so throughput is far below a GPU that holds the whole model. AirLLM is therefore aimed at people who want to run, inspect or experiment with a model their hardware could not otherwise load at all — a single hobbyist card, a laptop, or a Colab notebook — rather than at production serving, where vLLM, SGLang or TensorRT-LLM belong.

It is deliberately a thin layer over the Hugging Face ecosystem. You pass a Hugging Face repo id (or a local path) to a single `AutoModel.from_pretrained` call and then use `generate` as you would with Transformers, so most existing inference code needs almost no change. The project also ships an optional block-wise weight compression mode and supports Apple Silicon through MLX.

// What it does

• Layer-by-layer loading: only one transformer layer sits on the GPU at a time, so VRAM scales with layer size, not model size
• Per-expert streaming for sparse MoE models, loading only the experts a token actually routes to
• One `AutoModel.from_pretrained` entry point that auto-detects the architecture and accepts any Hugging Face repo id or local path
• Optional 4-bit or 8-bit block-wise weight compression, which the project reports as up to a 3x inference speed-up because less data is read from disk
• Works on Apple Silicon via MLX, and supports CPU inference as well as NVIDIA GPUs
• Configuration hooks for profiling, custom shard paths, gated-model tokens, prefetching, and deleting the original checkpoint to save disk

// Getting started

AirLLM installs as a single pip package. On first run it decomposes the model and saves it layer-wise, so make sure the Hugging Face cache directory has enough free disk space.

pip install airllm

from airllm import AutoModel

MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
# go bigger with the exact same one line:
# model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")  # 671B, runs in ~12GB

input_tokens = model.tokenizer(
    ["What is the capital of United States?"],
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False,
)

generation_output = model.generate(
    input_tokens["input_ids"].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True,
)

print(model.tokenizer.decode(generation_output.sequences[0]))

pip install -U bitsandbytes airllm

model = AutoModel.from_pretrained(
    "garage-bAInd/Platypus2-70B-instruct",
    compression="4bit",
)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when you want to run a model that is far too large for your GPU and can accept slow generation in exchange
• Use it to smoke-test or inspect the outputs of a frontier open-weight model on one consumer card before renting multi-GPU hardware
• Good for notebook and Colab experiments where the alternative is not running the model at all
• Skip it for production serving — a throughput engine such as vLLM or SGLang with enough VRAM will be orders of magnitude faster

// How AirLLM compares

AirLLM alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.