PowerInfer

// Overview

PowerInfer is a CPU/GPU inference engine for running large language models on a personal computer with a single consumer-grade GPU. It builds on the idea that LLM neuron activations follow a power-law: a small set of 'hot' neurons fire on almost every input, while most 'cold' neurons only activate for specific inputs.

To use this, PowerInfer keeps the frequently activated hot neurons on the GPU for fast access and computes the rarely used cold neurons on the CPU. This cuts GPU memory pressure and CPU-GPU data transfer, so a 24 GB card like an RTX 4090 can serve models that would otherwise need server-grade hardware.

It fits the local-runtime category alongside tools like llama.cpp. PowerInfer is a separate engine but reuses a similar workflow, works with ReLU-sparse GGUF models, and can also load standard llama.cpp weights for compatibility (without the speedup). It is aimed at developers who want low-latency local inference and serving rather than cloud APIs.

// What it does

• Locality-centric design: exploits sparse activation and the hot/cold neuron split for faster inference at lower resource cost
• Hybrid CPU/GPU execution: hot neurons run on the GPU, cold neurons on the CPU, balancing the workload across both
• Runs on consumer hardware: deeply optimized for low-latency inference on a single GPU
• Works with ReLU-sparse models including Falcon-40B, the Llama 2 family, ProSparse Llama 2, and Bamboo-7B
• Backward compatible: reuse most of the llama.cpp examples/ workflow such as server and batched generation
• Cross-platform: x86-64 with AVX2 on Linux and Windows (with or without NVIDIA GPU), plus Apple M chips on macOS (CPU only)

// Getting started

Clone the repo, install Python requirements, build with CMake for your hardware, then run inference with a PowerInfer GGUF model.

git clone https://github.com/Tiiny-AI/PowerInfer
cd PowerInfer
pip install -r requirements.txt

cmake -S . -B build -DLLAMA_CUBLAS=ON
cmake --build build --config Release

./build/bin/main -m ./ReluFalcon-40B-PowerInfer-GGUF/falcon-40b-relu.q4.powerinfer.gguf -n 128 -t 8 -p "Once upon a time"

./build/bin/main -m /PATH/TO/MODEL -n $output_token_count -t $thread_num -p $prompt --vram-budget $vram_gb

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Running a 40B-class model locally on a single 24 GB consumer GPU instead of renting server-grade hardware
• Low-latency local chat or generation on a developer workstation without sending data to a cloud API
• Serving or batch-generating with ReLU-sparse models using the llama.cpp-style examples workflow
• Experimenting with the hot/cold neuron offloading approach to fit larger models within a fixed VRAM budget

// How PowerInfer compares

PowerInfer alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.