Swiftlet

// Overview

Swiftlet is an inference runtime written in Swift with Metal GPU kernels, built for one specific trick: it keeps only the small dense core of a Mixture-of-Experts model resident in memory and streams the routed expert weights from storage on demand. Because a MoE model activates a fraction of its parameters per token, that turns a model far larger than the device's RAM into something an iPhone can hold open.

The runtime targets Qwen MoE models — Qwen3.6-35B and Qwen3-Next-80B in 4-bit and 8-bit variants — on Apple Silicon Macs running macOS 14 or later and iPhones running iOS 17 or later. Reported throughput spans roughly 1 token per second on an iPhone 17 up to about 19.5 tokens per second on an M4 Max, so the trade is explicit: the device stops being the limit on model size and starts being the limit on speed.

Under the hood it caches experts with an LFU eviction policy, compiles its Metal shaders at runtime, and validates its outputs against MLX reference implementations. Swiftlet is distributed as a Swift library, a set of command-line tools, an OpenAI-compatible server and an iOS app (Priv AI), all under the Apache-2.0 licence.

// What it does

• Streams routed MoE expert weights from storage while the dense core stays in memory
• Runs Qwen3.6-35B and Qwen3-Next-80B in 4-bit and 8-bit variants on Apple Silicon and on iPhone
• LFU expert cache keeps the hot experts resident and evicts the rest
• Metal shaders compiled at runtime, with outputs validated against MLX reference implementations
• Ships as a Swift library, CLI tools, an OpenAI-compatible server and the Priv AI iOS app
• Runs on macOS 14+ and iOS 17+, with no cloud call in the loop

// Getting started

Swiftlet builds with the Swift package manager and runs against a .qpack model file. Everything happens on-device.

git clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet
swift build -c release

.build/release/swiftlet chat ~/models/qwen3.6-35b.qpack "your prompt"

# 35B for phones and small Macs, 80B where storage bandwidth allows

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Run a large open-weight MoE model on a Mac that does not have enough RAM to hold it
• Put a 35B-class assistant on an iPhone with no network round-trip
• Build an iOS or macOS app on a Swift-native inference library instead of bridging to a C runtime
• Serve a local OpenAI-compatible endpoint from an Apple machine for development

// How Swiftlet compares

Swiftlet alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.