smolmaxxing

// Overview

smolmaxxing is a patch for llama.cpp that tries to run the biggest language model possible on the smallest hardware you own. It currently targets two devices with about 1 GB of RAM: a Xiaomi TV Box S (3rd Gen) with four Cortex-A55 cores and about 1 GB usable, running Android 14 without root, and a 2013 iPad Air with a two-core Apple A7 and about 700 MB usable, jailbroken on iOS 12.5.7. The project says every number it publishes was measured on the device itself.

The gains come from several tricks stacked together. Custom kernels are written per chip: a 1-bit lookup-table kernel, repacked 2-bit and ternary formats for the Cortex-A55, and versions for the Apple A7, which has no dot-product instructions. Memory use drops by removing rarely used vocabulary rows, and on the TV box a headless mode stops the launcher and Google services while a model runs. For mixture-of-experts models, popular experts stay in RAM and the rest are read from flash on demand, which lets 8B and 16B MoE models run in about 1 GB; dense models larger than RAM are streamed from flash.

The README reports Qwen3.5-2B at 7.7 tokens/s and the 16B Ling-mini-2.0 MoE at 5.1 tokens/s on the TV box, and the 1-bit Bonsai-1.7B at 24–26 tokens/s and Ling-mini-2.0 at 2.6–4.5 tokens/s on the iPad. Models need the patched llama.cpp; stock llama.cpp cannot load them. The code is MIT-licensed and the model weights keep their own licenses. Phones, single-board computers and routers are listed as planned devices.

// What it does

• Per-chip kernels: a 1-bit lookup-table kernel, repacked 2-bit and ternary formats for Cortex-A55, and Apple A7 paths without dot-product instructions
• Mixture-of-experts paging from flash, keeping popular experts in RAM, so 8B and 16B MoE models run in about 1 GB
• Streaming of dense models that are larger than RAM
• Vocabulary pruning that drops rarely used rows of the model's output layer to save memory
• Headless mode on the TV box that stops the launcher and Google services while a model runs
• A browser chat UI served from the TV box with live speed, model and config panels

// Getting started

The TV-box path needs network ADB enabled on the box and adb, zig, cmake, ninja, python3 and uv on your computer. Everything on the box lives in /data/local/tmp/llm. The iPad path is in ipad-air/README.md.

export ADB_SERIAL=<tv-ip>:5555
adb connect $ADB_SERIAL
git clone https://github.com/ggml-org/llama.cpp third_party/llama.cpp
cd third_party/llama.cpp && git checkout 8642541 && git apply ../../patches/llama.cpp-v0.8.0-tvbox.patch

tools/rebuild-models.sh 2b

tv-llm-ui/start.sh
tv-llm-ui/start.sh stop
sh /data/local/tmp/llm/tv-llm-mode.sh off

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Turn a cheap Android TV box into a small LLM server for your home network
• Give an old iPad a second life as an offline chat device
• Study which memory and kernel tricks matter most when RAM, not compute, is the limit
• Run a 16B mixture-of-experts model on hardware with about 1 GB of usable RAM

// How smolmaxxing compares

smolmaxxing alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.