Colibrì

// Overview

Colibrì is an inference engine written in pure C that runs very large Mixture-of-Experts (MoE) models on ordinary machines. Its central idea is that a MoE model does not have to fit in fast memory — it has to be placed. The dense part of the model (attention, shared experts, embeddings) stays resident in RAM at int4, while the thousands of routed experts live on disk and are streamed in only when the router asks for them. The project compares this to a JIT compiler: rather than loading everything up front, it watches which experts your workload actually routes to and stages those into the faster tiers.

The engine treats VRAM, RAM and NVMe as a single multitier hierarchy, with a per-layer LRU cache, a learned pinned hot-store recorded in a `.coli_usage` file, and one-layer-ahead prefetch to hide staging latency. The stated guarantee is that placement only ever changes speed, never semantics: less fast memory makes generation slower, but the router's decisions and the weights' precision stay the same. It also supports optional dual-SSD striping, local cluster mode with disk-backed expert workers, and CPU, CUDA, Metal and Vulkan backends.

Colibrì ships one C file per model family behind a single `coli` launcher, so the same `coli chat` / `coli serve` / `coli web` commands work across every supported model. The README documents eight families running today — GLM-5.2/5.3, GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6 and OLMoE. It is also explicitly an open research platform: the project publishes measured benchmark tables, keeps a list of open hypotheses, and asks contributors to publish negative results alongside wins.

// What it does

• Streams routed MoE experts from NVMe on demand while keeping the dense trunk resident in RAM at int4, so a 744B model can answer from roughly 9.9 GB of resident memory
• One memory hierarchy across VRAM, RAM and disk, with a per-layer LRU cache, a learned pinned hot-store and one-layer-ahead prefetch driven by measured routing heat
• Pure C engine with no BLAS, no Python at runtime and no GPU requirement; CPU, CUDA, Metal and Vulkan backends share the same runtime
• Optional dual-SSD striping that hashes each expert to one of two drives by measured bandwidth, with startup validation and automatic fallback to the primary drive
• Local cluster mode: a coordinator keeps token generation, routing and KV state local while disk-backed expert workers run routed FFNs on other machines
• A web dashboard (`coli web`) with live token metrics, a VRAM/RAM/disk tier bar, and Brain and Atlas pages that visualise expert routing
• One command line for every model family — `coli` reads the model's `config.json`, picks the matching engine binary and renders that family's chat template

// Getting started

You need two things: the program (a few hundred KB) and the model container. Prebuilt release archives cover Linux, macOS and Windows; only Python 3 is required, for the launcher and the optional API gateway.

mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info

git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh

# https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp

COLI_MODEL=/nvme/glm52_i4 ./coli chat
COLI_MODEL=/nvme/glm52_i4 ./coli plan
COLI_MODEL=/nvme/glm52_i4 ./coli doctor
./coli serve --model /nvme/glm52_i4

COLI_MODEL=/fast/glm52_i4 COLI_MODEL_MIRROR=/second/glm52_i4 ./coli chat

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Run a frontier-size Mixture-of-Experts model privately on a workstation or laptop instead of renting it behind an API
• Serve a large open-weight model from a single box over an OpenAI-compatible endpoint, with the expert set streamed from NVMe rather than held in VRAM
• Trade throughput for capability on hardware you already own, when a slow answer from a 744B model beats a fast answer from a small one
• Study MoE routing behaviour directly — the dashboard's Brain and Atlas pages render measured expert affinity for a running model

// How Colibrì compares

Colibrì alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.