TensorFold

// Overview

TensorFold is a Python server that runs language models on Apple Silicon Macs through MLX and on NVIDIA GPUs through CUDA, and exposes them through an OpenAI-compatible API. Both backends serve chat completions, completions, the OpenAI Responses API (`/v1/responses`) and the Anthropic Messages API (`/v1/messages`) at `http://127.0.0.1:8080/v1` by default, so existing SDKs and agents only need a new base URL.

Rather than one generic engine, each supported model family supplies its own kernels and its own draft verification. The project supports a fixed list of checkpoints, among them Nemotron 3.5 Lightning, Qwen3.8-27B, Qwen3.8 Flash Next, GLM-5.3-Flash, Gemma 4 26B-A4B and DeepSeek-V4-Flash, with drafts coming from a model's own MTP head, a DFlash or DSpark draft model, or copies of the prompt context. `tensorfold models` lists the families and checkpoints.

Its defining rule is exact decoding: a drafted token is accepted only when it equals the token the same engine would produce serially, and concurrent streams on the MLX engine must match their solo output. The README is explicit that exactness holds against the same engine, weights, runtime and settings, not between MLX and CUDA or between quantizations. On NVIDIA the CUDA kernels need compute capability 8.9 or newer (Ada, Hopper, Blackwell, including the DGX Spark's GB10); RTX 30 cards are refused at startup. The project is Apache-2.0 from version 0.6.0 (earlier releases were MIT) and requires Python 3.11 or newer.

// What it does

• OpenAI chat completions, completions and Responses plus Anthropic Messages endpoints on both the MLX and CUDA backends
• Exact speculative decoding: MTP heads, DFlash/DSpark draft models and context copies, with drafts accepted only when they match serial output (`"draft": false` per request to compare)
• Per-family kernels for MLX affine quantized checkpoints, plus NVFP4, FP8 and experimental EXL3 checkpoints on CUDA
• Memory-aware admission, shared decode rounds across concurrent requests (`--parallel`), retained conversation prefixes and on-disk prefix snapshots on MLX
• Optional image input with the `vision` extra and `--vision` for supported GLM-5.3-Flash and Qwen checkpoints
• Two-rank CUDA execution (`--tp 2`), a macOS launchd service manager (`tensorfold service`) and a live dashboard (`tensorfold tui`)

// Getting started

TensorFold needs Python 3.11 or newer and either an Apple Silicon Mac (MLX 0.32.2 or newer, installed by pip) or an NVIDIA GPU with compute capability 8.9 or newer. These steps follow the project's README and installation runbook.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install git+https://github.com/ashhart/TensorFold.git
tensorfold --version
tensorfold models

tensorfold info TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
tensorfold serve TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit --name local-model --context 8192

curl -fsS http://127.0.0.1:8080/health
curl -fsS http://127.0.0.1:8080/v1/models
curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128}'

docker run -it --gpus all --ipc=host --network host nvcr.io/nvidia/pytorch:26.07-py3
python -m pip install git+https://github.com/ashhart/TensorFold.git
tensorfold pull TensorFold/Qwen3.8-27B-MLX-4bit z-lab/Qwen3.8-27B-DFlash2
tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit --name local-model --host 0.0.0.0 --port 8080

tensorfold update --check
tensorfold update

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Serve a supported open-weight model such as Qwen3.8-27B or Nemotron 3.5 Lightning locally on an Apple Silicon Mac with drafted decoding that does not change the output
• Run the same models on a DGX Spark, RTX 40/50 series or Hopper/Blackwell GPU, or split one across two CUDA ranks
• Point OpenAI- or Anthropic-SDK clients and coding agents at a local backend by changing only the base URL
• Keep a model running as a login service on a Mac with `tensorfold service` and monitor it with `tensorfold tui`

// How TensorFold compares

TensorFold alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.