# InferSSD

> A llama.cpp-based engine that streams Mixture-of-Experts weights from SSD so models larger than your RAM or VRAM run locally

- Category: Models & Inference / Local Runtimes
- Language: C++
- Licence: Apache-2.0
- Repository: https://github.com/wici-ai/inferssd
- Website: https://wici.ai/wici-infer
- Canonical page: https://ai-tldr.dev/tools/inferssd/
- Updated: 2026-11-18

## In short

InferSSD is an inference engine from WiCi AI that runs large Mixture-of-Experts (MoE) models on machines whose RAM and VRAM are smaller than the model. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in C++ and licensed Apache-2.0. To get started: install on macOS or Linux.

## Overview

InferSSD is an inference engine from WiCi AI that runs large Mixture-of-Experts (MoE) models on machines whose RAM and VRAM are smaller than the model. It builds on llama.cpp and uses standard GGUF files as their publishers released them: the model's router picks the experts each token needs, and InferSSD indexes, reads, caches and executes those expert weights from the SSD instead of holding the whole model in memory. The README lists tested GGUFs for DeepSeek-V4.1-Flash-1017 (763B total), Muse-Spark-Open (816B total), Qwen3.8 Flash Next, Qwen3.6-35B-A3B and Gemma4 E4B.

Under the hood, source indexes locate each expert's byte range inside the original GGUF files, so indexing does not copy the model payload. Reads go through one read ring per operating system (io_uring on Linux, IoRing on Windows, reader threads with F_NOCACHE on macOS) into a RAM expert cache; with CUDA, or ROCm on AMD GPUs, a VRAM expert tier joins it, while on Metal and Vulkan a GPU lane takes a prompt's heavy experts and decode stays on the CPU. A shared placement policy measures costs while a pass runs and decides what to cache and prefetch.

It is used two ways: from the command line, where `inferssd cli` chats in the terminal and `inferssd serve` starts a browser chat plus an OpenAI-compatible API on port 8080, or through WiCi Infer, a free desktop app for macOS and Windows built on InferSSD that sets up the engine and downloads models without a terminal. The project publishes its benchmark tables against llama.cpp and Colibri on four test machines, with the methodology in its docs, and is licensed Apache-2.0.

## What it does

- Runs MoE models larger than RAM or VRAM by reading the routed experts a token needs from SSD, built on llama.cpp and ggml
- Uses standard GGUF files as published — source indexes point at expert byte ranges in the original files without copying the payload
- A RAM expert cache on every platform, plus a VRAM expert tier on CUDA and ROCm and a GPU lane for heavy prompt experts on Metal and Vulkan
- `inferssd serve` gives a browser chat and an OpenAI-compatible API at http://127.0.0.1:8080/v1; `inferssd cli` chats in the terminal
- One-line installers for macOS, Linux and Windows that need no sudo, with prebuilt packages on Apple silicon and Windows x64
- No memory setting to tune: the engine measures the machine on first run and uses the memory available

## Getting started

Prebuilt packages cover macOS 15+ on Apple silicon and Windows 10/11 x64. On Linux the installer builds from source, which needs Git, Python 3.9+, a C/C++ compiler, systemd with cgroup v2, and the CUDA toolkit or the ROCm HIP SDK. If you'd rather not use a terminal, the WiCi Infer desktop app sets the engine up for you.

1. Install on macOS or Linux

   The installer works in a private environment and uses no sudo. Follow the printed PATH instruction if it shows one.

```bash
curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh
```

2. Or install on Windows

   Run this in PowerShell. A machine with an NVIDIA GPU gets the CUDA package; every other machine gets the Vulkan package.

```powershell
irm https://github.com/wici-ai/inferssd/releases/latest/download/install.ps1 | iex
```

3. Chat with a model in the terminal

   `-hf` downloads the model from Hugging Face on the first run (to ~/.local/share/inferssd/models, or %LOCALAPPDATA%\inferssd\models on Windows) and measures your machine. Use `--model-dir` to put the files on a specific drive. Stop with Ctrl+C or `/exit`.

```bash
inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF
```

4. Serve a web UI and an OpenAI-compatible API

   Open http://127.0.0.1:8080 in a browser, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. `--host 0.0.0.0` exposes it on your network — the API has no authentication, so anyone who can reach the port can use it.

```bash
inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF
```

5. Run another quantization from a local folder

   For a GGUF that `-hf` doesn't fetch by default, download all of its files to one folder and pass the first shard with `-m`.

```bash
hf download unsloth/DeepSeek-V4.1-Flash-1017-GGUF --include "UD-IQ1_S/*" --local-dir ~/models/deepseek
inferssd serve -m ~/models/deepseek/UD-IQ1_S/DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf
```

## When to use it

- Run a several-hundred-billion-parameter open MoE model such as DeepSeek-V4.1-Flash or Muse-Spark-Open on a laptop, a single-GPU desktop or a DGX Spark
- Serve a large local model to existing apps through an OpenAI-compatible endpoint without renting GPUs
- Keep using the GGUF files you already downloaded for llama.cpp instead of converting to a separate container format
- Give non-technical users a local large-model chat through the WiCi Infer desktop app

## Version history

- 2026-11-18 v0.1.5: Adds a CUDA package and Authenticode-signed packages for Windows, runs the expert VRAM tier on AMD GPUs through ROCm, and loads more model files, including Unsloth's DeepSeek V4.1 GGUFs. Prompts now prefill in one pass on every platform.
