Cactus Compute · 2026-05-12 · notable
Cactus Compute Needle — 26M-Parameter Function-Calling Model Distilled From Gemini 3.1, MIT-Licensed, 6000 tok/s on Cactus Runtime
Cactus Compute released Needle, a 26-million-parameter encoder-decoder distilled from Gemini 3.1 for on-device tool calling. MIT-licensed weights and dataset; runs locally on Mac and PC at 6000 tok/s prefill, 1200 tok/s decode.
A 26M-parameter open model distilled from Gemini 3.1 that does nothing but call tools — small enough to run on phones, watches, and glasses.
Key specs
| Parameters | 26M |
|---|---|
| GitHub stars | 268 |
| Pretraining tokens | 200B |
| Pretraining hardware | 16x TPU v6e |
| Pretraining time | 27h |
| Post training tokens | 2B function-call |
| Post training time | 45min |
| Prefill throughput | 6000 tok/s |
| Decode throughput | 1200 tok/s |
What is it?
Needle is Cactus Compute's first standalone open-weights model: 26 million parameters, MIT-licensed, distilled from Gemini 3.1 for the single task of function calling and tool use. It targets resource-constrained devices — phones, watches, smart glasses — where shipping a 1B+ model is impractical. Cactus claims Needle beats FunctionGemma-270m, Qwen-0.6B, Granite-350m, and LFM2.5-350m on single-shot function-call accuracy despite being 10x to 13x smaller.
How does it work?
The model is an encoder-decoder with pure attention and no feed-forward layers: a 12-layer encoder with GQA (8 query heads, 4 KV heads), an 8-layer decoder with self-attention plus cross-attention, model dimension 512, ZCRMSNorm, and a SentencePiece vocab of 8192. Pretraining used 200B tokens over 27 hours on 16 TPU v6e chips; post-training added 2B tokens of function-call data in 45 minutes. The runtime side ships with Cactus' production engine clocking 6000 tok/s prefill and 1200 tok/s decode.
Why does it matter?
Function calling is the bottleneck for on-device agents — without it, edge assistants have to round-trip every tool invocation through a remote model. A 26M model that fits comfortably on a watch and beats much larger function-call baselines is a credible answer. The release also includes the dataset-generation pipeline, so developers can finetune Needle on custom tools locally.
Who is it for?
Developers shipping on-device agents to phones, watches, and embedded hardware
Try it
git clone https://github.com/cactus-compute/needle