What is InferSSD?
InferSSD is an inference engine from WiCi AI that runs large Mixture-of-Experts (MoE) models on machines whose RAM and VRAM are smaller than the model. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in C++ and licensed Apache-2.0. To get started: install on macOS or Linux.
Overview
InferSSD is an inference engine from WiCi AI that runs large Mixture-of-Experts (MoE) models on machines whose RAM and VRAM are smaller than the model. It builds on llama.cpp and uses standard GGUF files as their publishers released them: the model's router picks the experts each token needs, and InferSSD indexes, reads, caches and executes those expert weights from the SSD instead of holding the whole model in memory. The README lists tested GGUFs for DeepSeek-V4.1-Flash-1017 (763B total), Muse-Spark-Open (816B total), Qwen3.8 Flash Next, Qwen3.6-35B-A3B and Gemma4 E4B.
Under the hood, source indexes locate each expert's byte range inside the original GGUF files, so indexing does not copy the model payload. Reads go through one read ring per operating system (io_uring on Linux, IoRing on Windows, reader threads with F_NOCACHE on macOS) into a RAM expert cache; with CUDA, or ROCm on AMD GPUs, a VRAM expert tier joins it, while on Metal and Vulkan a GPU lane takes a prompt's heavy experts and decode stays on the CPU. A shared placement policy measures costs while a pass runs and decides what to cache and prefetch.
It is used two ways: from the command line, where `inferssd cli` chats in the terminal and `inferssd serve` starts a browser chat plus an OpenAI-compatible API on port 8080, or through WiCi Infer, a free desktop app for macOS and Windows built on InferSSD that sets up the engine and downloads models without a terminal. The project publishes its benchmark tables against llama.cpp and Colibri on four test machines, with the methodology in its docs, and is licensed Apache-2.0.
What it does
- Runs MoE models larger than RAM or VRAM by reading the routed experts a token needs from SSD, built on llama.cpp and ggml
- Uses standard GGUF files as published — source indexes point at expert byte ranges in the original files without copying the payload
- A RAM expert cache on every platform, plus a VRAM expert tier on CUDA and ROCm and a GPU lane for heavy prompt experts on Metal and Vulkan
- `inferssd serve` gives a browser chat and an OpenAI-compatible API at http://127.0.0.1:8080/v1; `inferssd cli` chats in the terminal
- One-line installers for macOS, Linux and Windows that need no sudo, with prebuilt packages on Apple silicon and Windows x64
- No memory setting to tune: the engine measures the machine on first run and uses the memory available
Getting started
Prebuilt packages cover macOS 15+ on Apple silicon and Windows 10/11 x64. On Linux the installer builds from source, which needs Git, Python 3.9+, a C/C++ compiler, systemd with cgroup v2, and the CUDA toolkit or the ROCm HIP SDK. If you'd rather not use a terminal, the WiCi Infer desktop app sets the engine up for you.
Install on macOS or Linux
The installer works in a private environment and uses no sudo. Follow the printed PATH instruction if it shows one.
curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | shOr install on Windows
Run this in PowerShell. A machine with an NVIDIA GPU gets the CUDA package; every other machine gets the Vulkan package.
irm https://github.com/wici-ai/inferssd/releases/latest/download/install.ps1 | iexChat with a model in the terminal
`-hf` downloads the model from Hugging Face on the first run (to ~/.local/share/inferssd/models, or %LOCALAPPDATA%\inferssd\models on Windows) and measures your machine. Use `--model-dir` to put the files on a specific drive. Stop with Ctrl+C or `/exit`.
inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUFServe a web UI and an OpenAI-compatible API
Open http://127.0.0.1:8080 in a browser, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. `--host 0.0.0.0` exposes it on your network — the API has no authentication, so anyone who can reach the port can use it.
inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUFRun another quantization from a local folder
For a GGUF that `-hf` doesn't fetch by default, download all of its files to one folder and pass the first shard with `-m`.
hf download unsloth/DeepSeek-V4.1-Flash-1017-GGUF --include "UD-IQ1_S/*" --local-dir ~/models/deepseek
inferssd serve -m ~/models/deepseek/UD-IQ1_S/DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.ggufCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Run a several-hundred-billion-parameter open MoE model such as DeepSeek-V4.1-Flash or Muse-Spark-Open on a laptop, a single-GPU desktop or a DGX Spark
- Serve a large local model to existing apps through an OpenAI-compatible endpoint without renting GPUs
- Keep using the GGUF files you already downloaded for llama.cpp instead of converting to a separate container format
- Give non-technical users a local large-model chat through the WiCi Infer desktop app
How InferSSD compares
InferSSD alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Ollama | ★ 188k | A developer-friendly tool that downloads and runs local LLMs from the terminal with a built-in OpenAI-compatible API. |
| llama.cpp | ★ 136k | A C/C++ inference engine that runs LLMs in the GGUF format on CPUs, Apple Silicon, and GPUs with low memory use. |
| GPT4All | ★ 77.3k | GPT4All is a free desktop app and Python client that runs large language models locally on your own computer, with no API calls or GPU required. |
| Colibrì | ★ 52.2k | A pure-C inference engine that keeps a Mixture-of-Experts model's dense trunk resident in RAM and streams its routed experts from disk, so 744B-2.8T models run on consumer hardware. |
| LocalAI | ★ 50.5k | A self-hosted server that exposes an OpenAI-compatible API for running text, vision, voice, and image models on local hardware. |
| Strata | ★ 45.9k | A one-click Windows and Linux engine that splits the 125B Qwen3.8-Flash-Next across GPU, RAM, CPU and SSD so it runs on a 12 GB gaming card, with OpenAI and Anthropic APIs on localhost. |
| Jan | ★ 45.9k | An open-source desktop app that runs LLMs fully offline as a ChatGPT-style assistant on your own computer. |
| InferSSD | ★ 1.3k | A llama.cpp-based engine that streams Mixture-of-Experts weights from SSD so models larger than your RAM or VRAM run locally |
