Cactus

// Overview

Cactus is a hybrid edge-cloud AI engine aimed at mobile devices and wearables. Rather than wrapping an existing runtime, it is built as four stacked layers that the project maintains itself: Cactus Quants, a rotation-and-codebook quantization scheme that takes weight tensors from 4-bit down to 1-bit; Cactus Kernels, hand-written ARM NEON SIMD kernels for matmul, attention, convolution, quantization, DSP and image processing; Cactus Graph, a zero-copy computation graph you can drive directly for tensor work; and Cactus Engine, which exposes OpenAI-compatible APIs for text, speech and vision on top of the rest.

The engine's C API covers chat completion, streaming, tool calling, transcription, embeddings, RAG and a vector index. Each generation returns telemetry alongside the text — time to first token, prefill and decode tokens per second, peak RAM, and a model confidence score. That confidence score drives the "hybrid" part: with cloud handoff enabled, a request the local model is not confident about can be routed to a hosted model instead, and the response marks whether the handoff happened.

Cactus is source-available rather than OSI open source. Its licence grants free use to individuals, students, educational institutions, registered non-profits, and organisations with both under $2M in total funding and under $2M in gross annual revenue; anyone above those thresholds needs a commercial licence from Cactus Compute, Inc. Bindings ship for Swift, Kotlin, Flutter, React Native, Python and Rust.

// What it does

• Rotation-and-codebook quantization (CQ) from 4-bit down to 1-bit, including the mixed-precision CQ3.26 and CQ2.54 formats, with published accuracy tables per bit width
• ARM NEON SIMD kernels written for the stack, plus CPU and Metal backends selected automatically
• OpenAI-compatible engine API covering chat, streaming, tool calling, transcription, embeddings, vision and a built-in vector index for RAG
• Automatic cloud handoff: requests below a configurable confidence threshold are routed to a hosted model, and the response reports whether that happened
• `cactus convert` turns a Hugging Face model into CQ weights — Liquid, Gemma, Whisper, Parakeet and Qwen families are the tested ones — and `cactus download` pulls pre-converted bundles
• Bindings for Swift, Kotlin, Flutter, React Native, Python and Rust, so the same engine backs a native iOS app and an Android one

// Getting started

The quickest look is the Homebrew CLI on a Mac; building from the repository gives you the full command set and the language bindings.

brew install cactus-compute/cactus/cactus
cactus run

sudo apt-get install python3.12 python3.12-venv python3-pip cmake \
  build-essential libcurl4-openssl-dev

git clone https://github.com/cactus-compute/cactus && cd cactus
source ./setup

# chat with a model, choosing the quantization width
cactus run google/gemma-4-e2b-it --bits 4 --backend metal

# vision and audio inputs use the same command
cactus run <model> --image photo.jpg
cactus transcribe --file meeting.wav --language en

# convert your own checkpoint
cactus convert <hf-name> --bits 3

# OpenAI-compatible server on 127.0.0.1:8080
cactus serve <model> --port 8080 --confidence-threshold 0.7

#include "cactus_engine.h"

cactus_model_t model = cactus_init(
    "path/to/weight/folder",
    "path to txt or dir of txts for auto-rag",
    false
);

const char* messages = R"([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Summarise this in one line."}
])";

char response[4096];
int result = cactus_complete(
    model, messages, response, sizeof(response),
    R"({"max_tokens": 50})",
    nullptr, nullptr, nullptr, nullptr, 0
);

cactus benchmark
cactus benchmark --ios
cactus benchmark --android

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Ship an assistant inside a mobile app that answers offline and only reaches for a hosted model when the local one is unsure
• Fit a text, vision or speech model into a wearable or robot's memory budget by trading accuracy against bit width with measured numbers per task
• Give an iOS and an Android app the same inference behaviour through the Swift, Kotlin, Flutter or React Native bindings
• Run on-device tool calling with a small dedicated model rather than round-tripping every function call to a server

// How Cactus compares

Cactus alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.