DwarfStar

// Overview

DwarfStar — the repository is antirez/ds4 — is a small native inference engine aimed at running a handful of excellent open-weight models on hardware people actually own. It targets DeepSeek V4 Flash (including the experimental vision model), DeepSeek V4 PRO, GLM 5.2, GLM 5.3 and GLM 5.3 Flash. It is deliberately not a general GGUF runner: you use the GGUF files the project itself produces, which are part of the project.

The primary target is Metal on Macs with 96 GB or more; smaller machines can fall back to SSD streaming, which is also how you run a full (non-Flash) GLM 5.x on a 128 GB system. NVIDIA CUDA is supported with the DGX Spark as the main goal, and multi-GPU setups that other backends do not cover — the project reports about 126 tokens/s aggregate generation across 16 sessions on an eight-L40S Flash setup. ROCm covers Strix Halo machines such as the Framework Desktop. Two 128 GB Macs joined over RDMA can run 4-bit DeepSeek Flash or GLM 5.3 Flash with tensor parallelism, and pipeline parallelism can glue several systems together to pool their RAM.

The repo ships more than a library: an interactive CLI (`ds4`), an HTTP server (`ds4-server`) and a native coding agent (`ds4-agent`) that runs inference in-process and uses each model's own tool format. Model loading, prompt rendering, tool calls, KV state, the server and the agent are built and tested together. The author is explicit that the project is beta and fast-moving, that it is developed with heavy assistance from AI coding agents, and that it exists thanks to llama.cpp and GGML — whose quant layouts, CPU quant logic and some kernels are retained here under the MIT license.

// What it does

• Native C engine specialised for DeepSeek V4 Flash/PRO and GLM 5.2 / 5.3 / 5.3 Flash, using GGUF files the project builds itself
• Metal on Apple Silicon as the primary backend, plus CUDA (including Ada Lovelace and L40S multi-GPU) and ROCm on Strix Halo
• SSD streaming so models larger than available RAM still run at a usable speed
• Tensor parallelism over RDMA between two machines, and pipeline parallelism to pool RAM across several systems
• Three front ends built and tested together: the ds4 interactive CLI, ds4-server on port 8000, and the in-process ds4-agent coding agent
• Session management for the agent — /save, /list, /switch, /del and /strip against KV snapshots in ~/.ds4/kvcache
• Bundled tooling and data for GGUF conversion, imatrix, quality and speed measurement

// Getting started

Clone the repo and pick the build for your hardware; each platform guide in docs/ covers prerequisites and memory sizing. Model support is intentionally opportunistic — a model can be dropped when a better replacement arrives.

git clone https://github.com/antirez/ds4.git
cd ds4

make               # Metal on Apple Silicon
make cuda-spark    # DGX Spark
make strix-halo    # Strix Halo / Framework Desktop
make cuda-generic  # one or more CUDA cards, including Ada/L40S

./download_model.sh ds4f-q2

./ds4
./ds4 -p "Explain Redis streams in one paragraph."
./ds4-agent
./ds4-server --ctx 32768

./ds4-server --ctx 32768
# then configure your client against http://127.0.0.1:8000

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Run DeepSeek V4 Flash or GLM 5.3 Flash locally on a 96–128 GB Mac, or stream from SSD on a smaller one
• Serve a multi-user local LLM endpoint from several CUDA cards, including generations other backends have dropped
• Pool two machines over RDMA or several over pipeline parallelism to fit a model no single box holds
• Use a coding agent that runs inference in-process, keeping token history and model state together

// How DwarfStar compares

DwarfStar alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.