Edge0 · 2026-09-16 · major
Edge0 report — how a 35B MoE runs from an SSD in 2.9 GB
Edge0 is an open-source framework that runs a 35B mixture-of-experts model on Apple Silicon by streaming experts from the SSD, keeping peak active memory near 2.9 GB. A new technical report explains the trained prerouter that makes it work.
A 35B mixture-of-experts model that lives on your SSD and pages in only the experts each token needs.
Quick facts
| Maker | Edge0 |
|---|---|
| Model | Edge0-35B-A3B-preview |
| Architecture | 256 experts, 4 active per token |
| Peak active memory | ~2.9 GB |
| Decode speed | 14.9–17.7 tokens/s |
| Platform | macOS on Apple Silicon (MLX) |
| License | Apache-2.0 |
Benchmarks
What is it?
Edge0 keeps a 35B model's expert weights in memory-mapped files on disk instead of loading them into RAM up front. The Apache-2.0 framework serves the Edge0-35B-A3B-preview checkpoint — built on Qwen3.6-35B-A3B, with 4 of its 256 experts active per token — at roughly 2.9 GB of peak active memory on a 24 GB machine. A technical report published on 16 September 2026 documents how the system does it.
How does it work?
The core trick is a prerouter: a small per-layer head predicts the next layer's expert routing one token ahead, so those experts start loading while the current layer is still computing. Weight streaming overlaps with compute instead of stalling on it. A separate unmerged recovery LoRA, trained on the student path, wins back accuracy lost to 4-bit quantization and to the changed routing.
Why does it matter?
Running a 35B-class model normally means renting a GPU or buying a large machine. Edge0's numbers point at a laptop instead, at mid-teens tokens per second on the 35B tier, plus a smaller edge0-8b tier for lighter hardware. The costs are accuracy — the report measures an average gap against the full-precision teacher — and reach, since the backend is MLX on Apple Silicon and CUDA support is still on the roadmap.
Who is it for?
people running local models on Apple Silicon
Frequently asked questions
- Can Edge0 run on an NVIDIA GPU?
- Edge0 ships with an MLX backend only, so today it runs on macOS with Apple Silicon, from M1 through M4. The project lists a CUDA backend on its roadmap and keeps the backend behind shared core abstractions so other platforms can plug into the same code, but no CUDA build exists in the current release.
- How much accuracy does Edge0 give up against the full-precision model?
- Edge0's int4 35B checkpoint averages 79.2 across the five benchmarks on its model card, against 83.2 for the fp16 Qwen3.6-35B-A3B model it is built on — an average gap of 3.9 points. The recovery LoRA exists to narrow exactly that gap, and the report presents the remainder as accuracy traded for fitting on consumer hardware.
- Is there a smaller Edge0 model?
- Yes. The edge0-8b tier uses 128 experts with 8 active per token, takes about 4.2 GB on disk and peaks near 1.0 GB of active memory, decoding at 23.9–25.3 tokens per second. It suits machines that cannot spare the roughly 23 GB of disk that the 35B tier's 4-bit weights need.
- How do I call an Edge0 model from my own code?
- Edge0 exposes an OpenAI-style HTTP endpoint. Running `edge0 serve edge0-35b` starts a local server on port 8000 that answers POST requests to /v1/chat/completions, so client code that already speaks the chat-completions format can point at the local machine. There is also `edge0 demo` for a quick check and `edge0 chat` for single prompts.
Try it
edge0 chat edge0-35b --prompt "Explain streaming inference in one sentence."