kimi-k3-in-c

// Overview

kimi-k3-in-c is a single-purpose inference engine: it runs Kimi K3 — 2.78 trillion parameters, a 1.56 TB checkpoint — in portable C99, with no BLAS, no framework and no GPU. The measured peak resident set for a generation run is 8.24 GB, which is the point of the project: memory stops being a floor you must clear and becomes a dial that only changes speed. On the author's published ladder the same prompt takes about 26.5 s per token on an 8 GB laptop and about 5.6 s per token on a 128 GB-plus machine, with byte-identical output at every size.

It gets there by streaming. Routed expert weights already ship at half a byte each, the trunk is packed so each of the 93 layers is one read, an LRU cache keeps hot experts in memory, and the KDA and MLA attention paths keep the per-token state from growing. Storage, not RAM, is the real gate: you need roughly 1.7 TB free for the checkpoint plus the 109 GB packed trunk, ideally on a fast local disk. The reference platform is Linux x86-64 with AVX2 and FMA; macOS/arm64 and Windows via MSYS2 MinGW-w64 also build and pass the test gates.

The repository takes correctness seriously enough to be usable as a reference. `make test` runs a weightless gate ladder — teacher forcing, greedy decode, and incremental decode with KV cache and carried KDA state — against a PyTorch reference over a 13-layer model built from the same tensor graph, in under a minute, with no checkpoint, no network and no Python. A `k3-doctor.sh` script checks your toolchain, sizes your RAM to a preset and measures your disk the way the engine reads it, and the download script verifies the checkpoint byte-exactly per shard so a partial download fails loudly instead of producing wrong tokens.

// What it does

• Portable C99 engine — seven C files, a compiler and OpenMP; no BLAS, no framework, no GPU
• Streams the checkpoint from disk so an 8 GB machine runs the full 2.78T-parameter model; more RAM buys speed, never different output
• Memory presets (laptop, server, …) sized to the machine by the bundled `k3-doctor.sh`
• LRU expert cache with a recorded trace you can replay to pick a capacity
• Weightless test suite: three gates against a PyTorch reference, no checkpoint or network needed
• Byte-exact per-shard checkpoint verification in the download script
• Builds with GNU make or CMake/ctest; Linux x86-64 reference, macOS/arm64 and Windows MinGW-w64 also supported
• Base model — no chat template, so output continues the prompt rather than replying to it

// Getting started

You can build and fully test the engine in about a minute without downloading anything. Requirements for a real run: AVX2 + FMA CPU, GCC ≥ 9 or Clang ≥ 10, ~1.7 TB free storage, and Python 3.9+ for the download and packing tools.

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c

make -j
make test

./scripts/k3-doctor.sh

export HF_TOKEN=hf_your_token_here
./scripts/download-model.sh ~/k3model
./scripts/pack-trunk.sh ~/k3model ~/k3trunk

./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
         --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it to run a frontier-scale open model on hardware you already own, when latency does not matter but capability does
• Reach for it when you need an auditable, dependency-free inference path — seven C files you can read end to end
• Reach for it as a correctness reference: the weightless gate ladder checks an implementation against PyTorch without a checkpoint
• Reach for it to study how expert streaming, trunk packing and cache capacity trade memory against tokens per second

// How kimi-k3-in-c compares

kimi-k3-in-c alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.