edge0

// Overview

edge0 is an open-source inference framework for sparse Mixture-of-Experts language models, built around one idea: a MoE model only uses a small slice of its parameters on any given token, so the rest does not need to sit in memory. edge0 keeps expert weights on disk, memory-maps them and reads them on demand, which means peak memory is bounded by the active expert set rather than by the total parameter count. The project's headline claim for its 35B tier is roughly 2.9 GB of peak active memory.

Streaming weights from storage normally means stalling the forward pass while the next expert loads, and that is what the other two mechanisms address. A trained **prerouter** head predicts expert routing one step ahead so loads overlap computation instead of blocking it — the README reports up to +59% decode throughput, and says the gain grows with storage latency, model size and routed width. **Recover-LoRA** handles the quality side: the int4 base is frozen and LoRA adapters are trained by distillation from the full-precision teacher, recovering most of the quantization loss. The adapters stay unmerged, so one read-only base can serve several adapter sets.

The framework ships two end-to-end tiers rather than raw weights alone — each is a checkpoint plus the LoRA and prerouter adapters trained for it, packaged in a single directory so `edge0 serve` runs the trained pipeline out of the box. `edge0-35b` is built on Qwen3.6-35B-A3B (4-bit, 40 layers, 256 experts) and `edge0-8b` on the Ling 3.0 bailing hybrid (4-bit, 24 layers, 128 experts). The project benchmarks both against their fp16 bases with OpenCompass under identical settings and reports an average loss of 3.9 points for the 35B tier and 2.8 for the 8B.

Backends are isolated by design. All MLX code lives under `edge0/backends/mlx/` and the core — model specs, prerouter, streaming expert pool, server — depends only on a backend facade, with `backends/cuda/` reserved as a slot for a future implementation. As of this writing the MLX backend is the only one shipped, so edge0 runs on macOS with Apple Silicon; CUDA is on the roadmap. The project is Apache-2.0 and labels the two published checkpoints as preview releases.

// What it does

• SSD expert offload: expert weights are streamed from storage on demand, so peak memory tracks the active set rather than the parameter count
• Prerouter: a trained head predicts routing one step ahead so expert loads overlap the forward pass — the README reports up to +59% decode throughput
• Recover-LoRA: adapters distilled from the full-precision teacher onto a frozen int4 base, recovering most of the 4-bit quantization loss
• Adapters stay unmerged as safetensors with provenance metadata, so one read-only base serves multiple adapter sets and upgrading swaps adapter files only
• transformers-style API — AutoModel / AutoConfig / AutoEngine resolve the tier from the model name
• OpenAI-compatible server: `edge0 serve` exposes /v1/chat/completions
• Backend facade with all MLX code isolated under edge0/backends/mlx/, leaving a reserved slot for a CUDA backend

// Getting started

edge0 needs Python 3.10+ (3.12 recommended) and, for the MLX backend it ships today, macOS on Apple Silicon. Install the package, download one of the two published model directories, then point the CLI at it.

python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'

.venv/bin/python scripts/fetch_models.py --tier edge0-35b --target-dir models

# or with the Hugging Face CLI directly:
.venv/bin/huggingface-cli download Edge0/Edge0-35B-A3B-preview --local-dir models/edge0-35b

edge0 demo models/edge0-35b

edge0 serve models/edge0-35b

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'

from edge0 import AutoEngine
from edge0.server.chat import ChatMessage, ChatRequest, ChatSession

engine = AutoEngine.from_pretrained("/path/to/model")
req = ChatRequest(
    model=engine.name,
    messages=[ChatMessage(role="user", content="Hello!")],
    max_tokens=64,
)
tokens, meta = ChatSession(engine, req).run()
print(engine._tok.decode(tokens))
engine.close()

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Running a 35B-class MoE model on an Apple Silicon laptop where the full weights would never fit in unified memory
• Serving a local OpenAI-compatible endpoint for apps that should not send prompts to a hosted API
• Trading disk for RAM on memory-constrained hardware, accepting streaming latency in exchange for a much larger model
• Experimenting with quantization recovery: comparing a plain int4 base against the same base with Recover-LoRA adapters attached

// How edge0 compares

edge0 alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.