█

AI/TLDR

Buy me a coffee
Buy me a coffee

InferSSD

A llama.cpp-based engine that streams Mixture-of-Experts weights from SSD so models larger than your RAM or VRAM run locally

Local RuntimesOpen source
Latest
v0.1.5
Updated
18 Nov 2026
Language
C++
License
Apache-2.0

What's new

v0.1.518 Nov 2026

Adds a CUDA package and Authenticode-signed packages for Windows, runs the expert VRAM tier on AMD GPUs through ROCm, and loads more model files, including Unsloth's DeepSeek V4.1 GGUFs. Prompts now prefill in one pass on every platform.

IN SHORT

What is InferSSD?

InferSSD is an inference engine from WiCi AI that runs large Mixture-of-Experts (MoE) models on machines whose RAM and VRAM are smaller than the model. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in C++ and licensed Apache-2.0. To get started: install on macOS or Linux.

Updated Markdown version ↗

Overview

InferSSD is an inference engine from WiCi AI that runs large Mixture-of-Experts (MoE) models on machines whose RAM and VRAM are smaller than the model. It builds on llama.cpp and uses standard GGUF files as their publishers released them: the model's router picks the experts each token needs, and InferSSD indexes, reads, caches and executes those expert weights from the SSD instead of holding the whole model in memory. The README lists tested GGUFs for DeepSeek-V4.1-Flash-1017 (763B total), Muse-Spark-Open (816B total), Qwen3.8 Flash Next, Qwen3.6-35B-A3B and Gemma4 E4B.

Under the hood, source indexes locate each expert's byte range inside the original GGUF files, so indexing does not copy the model payload. Reads go through one read ring per operating system (io_uring on Linux, IoRing on Windows, reader threads with F_NOCACHE on macOS) into a RAM expert cache; with CUDA, or ROCm on AMD GPUs, a VRAM expert tier joins it, while on Metal and Vulkan a GPU lane takes a prompt's heavy experts and decode stays on the CPU. A shared placement policy measures costs while a pass runs and decides what to cache and prefetch.

It is used two ways: from the command line, where `inferssd cli` chats in the terminal and `inferssd serve` starts a browser chat plus an OpenAI-compatible API on port 8080, or through WiCi Infer, a free desktop app for macOS and Windows built on InferSSD that sets up the engine and downloads models without a terminal. The project publishes its benchmark tables against llama.cpp and Colibri on four test machines, with the methodology in its docs, and is licensed Apache-2.0.

What it does

  • Runs MoE models larger than RAM or VRAM by reading the routed experts a token needs from SSD, built on llama.cpp and ggml
  • Uses standard GGUF files as published — source indexes point at expert byte ranges in the original files without copying the payload
  • A RAM expert cache on every platform, plus a VRAM expert tier on CUDA and ROCm and a GPU lane for heavy prompt experts on Metal and Vulkan
  • `inferssd serve` gives a browser chat and an OpenAI-compatible API at http://127.0.0.1:8080/v1; `inferssd cli` chats in the terminal
  • One-line installers for macOS, Linux and Windows that need no sudo, with prebuilt packages on Apple silicon and Windows x64
  • No memory setting to tune: the engine measures the machine on first run and uses the memory available

Getting started

Prebuilt packages cover macOS 15+ on Apple silicon and Windows 10/11 x64. On Linux the installer builds from source, which needs Git, Python 3.9+, a C/C++ compiler, systemd with cgroup v2, and the CUDA toolkit or the ROCm HIP SDK. If you'd rather not use a terminal, the WiCi Infer desktop app sets the engine up for you.

Install on macOS or Linux

The installer works in a private environment and uses no sudo. Follow the printed PATH instruction if it shows one.

bashbash
curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh

Or install on Windows

Run this in PowerShell. A machine with an NVIDIA GPU gets the CUDA package; every other machine gets the Vulkan package.

powershellpowershell
irm https://github.com/wici-ai/inferssd/releases/latest/download/install.ps1 | iex

Chat with a model in the terminal

`-hf` downloads the model from Hugging Face on the first run (to ~/.local/share/inferssd/models, or %LOCALAPPDATA%\inferssd\models on Windows) and measures your machine. Use `--model-dir` to put the files on a specific drive. Stop with Ctrl+C or `/exit`.

bashbash
inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

Serve a web UI and an OpenAI-compatible API

Open http://127.0.0.1:8080 in a browser, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. `--host 0.0.0.0` exposes it on your network — the API has no authentication, so anyone who can reach the port can use it.

bashbash
inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

Run another quantization from a local folder

For a GGUF that `-hf` doesn't fetch by default, download all of its files to one folder and pass the first shard with `-m`.

bashbash
hf download unsloth/DeepSeek-V4.1-Flash-1017-GGUF --include "UD-IQ1_S/*" --local-dir ~/models/deepseek
inferssd serve -m ~/models/deepseek/UD-IQ1_S/DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Run a several-hundred-billion-parameter open MoE model such as DeepSeek-V4.1-Flash or Muse-Spark-Open on a laptop, a single-GPU desktop or a DGX Spark
  • Serve a large local model to existing apps through an OpenAI-compatible endpoint without renting GPUs
  • Keep using the GGUF files you already downloaded for llama.cpp instead of converting to a separate container format
  • Give non-technical users a local large-model chat through the WiCi Infer desktop app

How InferSSD compares

InferSSD alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
Ollama★ 188kA developer-friendly tool that downloads and runs local LLMs from the terminal with a built-in OpenAI-compatible API.
llama.cpp★ 136kA C/C++ inference engine that runs LLMs in the GGUF format on CPUs, Apple Silicon, and GPUs with low memory use.
GPT4All★ 77.3kGPT4All is a free desktop app and Python client that runs large language models locally on your own computer, with no API calls or GPU required.
Colibrì★ 52.2kA pure-C inference engine that keeps a Mixture-of-Experts model's dense trunk resident in RAM and streams its routed experts from disk, so 744B-2.8T models run on consumer hardware.
LocalAI★ 50.5kA self-hosted server that exposes an OpenAI-compatible API for running text, vision, voice, and image models on local hardware.
Strata★ 45.9kA one-click Windows and Linux engine that splits the 125B Qwen3.8-Flash-Next across GPU, RAM, CPU and SSD so it runs on a 12 GB gaming card, with OpenAI and Anthropic APIs on localhost.
Jan★ 45.9kAn open-source desktop app that runs LLMs fully offline as a ChatGPT-style assistant on your own computer.
InferSSD★ 1.3kA llama.cpp-based engine that streams Mixture-of-Experts weights from SSD so models larger than your RAM or VRAM run locally