# Cactus Compute Needle — 26M-Parameter Function-Calling Model Distilled From Gemini 3.1, MIT-Licensed, 6000 tok/s on Cactus Runtime

> A 26M-parameter open model distilled from Gemini 3.1 that does nothing but call tools — small enough to run on phones, watches, and glasses.

- Org: Cactus Compute
- Date: 2026-05-12
- Category: model, repo
- Source: https://github.com/cactus-compute/needle
- Canonical page: https://ai-tldr.dev/releases/cactus-compute-needle-26m-function-call/

## In short

Cactus Compute released Needle, a 26-million-parameter encoder-decoder distilled from Gemini 3.1 for on-device tool calling. MIT-licensed weights and dataset; runs locally on Mac and PC at 6000 tok/s prefill, 1200 tok/s decode.

## What it is

Needle is Cactus Compute's first standalone open-weights model: 26 million parameters, MIT-licensed, distilled from Gemini 3.1 for the single task of function calling and tool use. It targets resource-constrained devices — phones, watches, smart glasses — where shipping a 1B+ model is impractical. Cactus claims Needle beats FunctionGemma-270m, Qwen-0.6B, Granite-350m, and LFM2.5-350m on single-shot function-call accuracy despite being 10x to 13x smaller.

## How it works

The model is an encoder-decoder with pure attention and no feed-forward layers: a 12-layer encoder with GQA (8 query heads, 4 KV heads), an 8-layer decoder with self-attention plus cross-attention, model dimension 512, ZCRMSNorm, and a SentencePiece vocab of 8192. Pretraining used 200B tokens over 27 hours on 16 TPU v6e chips; post-training added 2B tokens of function-call data in 45 minutes. The runtime side ships with Cactus' production engine clocking 6000 tok/s prefill and 1200 tok/s decode.

## Why it matters

Function calling is the bottleneck for on-device agents — without it, edge assistants have to round-trip every tool invocation through a remote model. A 26M model that fits comfortably on a watch and beats much larger function-call baselines is a credible answer. The release also includes the dataset-generation pipeline, so developers can finetune Needle on custom tools locally.

## Who it's for

Developers shipping on-device agents to phones, watches, and embedded hardware

## Try it

git clone https://github.com/cactus-compute/needle

## Links

- [GitHub Repo](https://github.com/cactus-compute/needle)
- [Hugging Face](https://huggingface.co/cactus-compute/Needle)
- [Cactus Site](https://cactuscompute.com)