TurboFieldfare

// Overview

TurboFieldfare is a single-model inference runtime written in Swift and Metal for Apple Silicon Macs. Rather than loading the whole 14.3 GB Gemma 4 26B-A4B checkpoint into memory, it keeps the shared 1.35 GB core and an FP16 KV cache resident and streams only the experts a token actually routes to from SSD. The README reports about 2 GB of weights plus a 4K KV cache in memory, which is what lets the model run on a Mac with 8 GB of RAM.

It is deliberately not a wrapper around MLX or llama.cpp — the kernels, the streaming installer, the CLI and the Mac app are all project code. Weights are stored as MLX affine 4-bit with group size 64, an 8-bit router, and 4-bit shared and routed experts, in a `.gturbo` model directory produced by the bundled repacker. The project publishes measured decode figures rather than estimates: 5.1–6.3 tokens per second on an 8 GB M2 MacBook Air and 31–35 on a 24 GB M5 Pro, with a community benchmark page collecting results from other machines.

The Swift package exposes six products: the `TurboFieldfare` runtime library, a native Mac app, a one-shot decode service the app owns the model through, a CLI for chat and raw completion, an experimental loopback OpenAI-compatible Chat Completions server, and the streaming repacker/installer. They share the same model directory, but only one model-owning product should run at a time. It targets macOS 26 with Metal 4 and Swift 6.2, and is Apache-2.0 licensed.

// What it does

• Runs Gemma 4 26B-A4B in about 2 GB of RAM by streaming routed experts from SSD instead of loading the full 14.3 GB checkpoint
• Custom Swift + Metal kernels rather than a wrapper around MLX or llama.cpp
• Native Mac app, a CLI, and an experimental loopback OpenAI-compatible Chat Completions server over one `.gturbo` model directory
• Streaming installer and repacker that fetches and converts the pinned model, with an install verifier
• Published, measured decode benchmarks (5.1–6.3 tok/s on an 8 GB M2 Air, 31–35 tok/s on a 24 GB M5 Pro) plus community-submitted results
• MLX affine 4-bit weights with an 8-bit router, and an optional image pack for multimodal input

// Getting started

TurboFieldfare builds with Swift Package Manager and downloads the pinned model on first run. It needs an Apple Silicon Mac on macOS 26 or later.

git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac

swift build -c release
.build/release/TurboFieldfareCLI
.build/release/TurboFieldfareServer

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when you want a 26B-class model on a Mac whose RAM could never hold the full checkpoint
• Reach for it when you want local inference on an 8 GB M-series machine and are willing to trade tokens per second for memory
• Reach for it when you want an OpenAI-compatible endpoint on localhost without running a server stack
• Reach for it when you are studying how expert streaming, kernel design and I/O overlap interact — the repo publishes 103 measured experiments

// How TurboFieldfare compares

TurboFieldfare alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.