Slotstream

// Overview

Slotstream is a local inference engine for Apple Silicon Macs built around one problem: Qwen3.8-Flash-Next is a 125B-parameter mixture-of-experts model that occupies 105 GB on disk at 4-bit, and most Macs do not have that much memory. Instead of shrinking the model, Slotstream changes how it is held: the 3.8 GB trunk stays resident, and the experts are streamed from SSD through a fixed pool of slots sized to what the machine actually has free. Engine start takes about two seconds because only the trunk loads.

It ships as a single Swift binary with no Python dependency, and it speaks two dialects that existing tooling already knows — the Ollama API and the OpenAI chat API — on port 11434, so clients such as Open WebUI and the OpenAI SDKs work against it unchanged. Capabilities the two dialects define but Slotstream does not implement (tools, JSON-schema output, logprobs) return an explicit 400 rather than being silently dropped.

The engine is MLX and Metal and the memory design assumes unified memory, so it is Apple Silicon only by design rather than by omission: experts are read from SSD straight into memory the GPU already addresses. The project documents its own measurements, including the auto-sizing table produced by `slotstream doctor --sim-ram N` and the caveat that the table has no term for SSD read speed — a base-storage Mac mini M2 measured well below the projected figure.

// What it does

• Streams mixture-of-experts weights from SSD through a fixed slot pool, keeping only the 3.8 GB trunk resident so the engine starts in about two seconds
• Auto-sizes its memory pool against what the machine really has free, and gives memory back when other apps want it; the plan is printed at startup
• Serves the Ollama (/api/chat, /api/generate) and OpenAI (/v1) chat subsets on port 11434, tested against Open WebUI and the OpenAI SDKs
• Single Swift binary — no Python runtime — installed by a one-line script or built from source with Command Line Tools
• Image input across every dialect: Ollama's images array, OpenAI image_url parts, and --image on the command line, backed by a 0.9 GB vision tower loaded on first use
• Resumable, sha256-verified weight downloads over eight parallel connections, with a doctor command that checks the machine before anything is transferred

// Getting started

Slotstream needs Apple Silicon, macOS 14 or later, and roughly 110 GB of free disk for the weights. Check the machine first, then install, pull, and serve.

curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh

slotstream doctor

slotstream run --prompt "why is the sky blue?"

slotstream serve

curl localhost:11434/api/chat -d '{
  "model": "qwen3.8-flash-next:4bit",
  "messages": [{"role": "user", "content": "hello"}]
}'

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Run a frontier-size open-weight MoE model locally on a Mac whose RAM is far below the model's on-disk size
• Point an existing Ollama- or OpenAI-compatible client (Open WebUI, an SDK script) at a local endpoint without changing the client
• Keep prompts and documents on the machine when a hosted API is not an option, at the cost of SSD bandwidth rather than RAM
• Measure what a specific Mac can actually sustain before committing to a 105 GB download, using the doctor command's simulated-RAM plan

// How Slotstream compares

Slotstream alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.