# llama-modchip

> llama.cpp kernel patches and 2-bit quant formats that run the 284B DeepSeek-V4-Flash MoE model on a 64 GB machine with no GPU

- Category: Models & Inference / Local Runtimes
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/synchopate/llama-modchip
- Canonical page: https://ai-tldr.dev/tools/llama-modchip/
- Updated: 2026-11-04

## In short

llama-modchip is a set of ggml/llama.cpp kernel patches, quantization tools and runtime utilities for running large sparse Mixture-of-Experts models on an ordinary CPU. Its headline target is DeepSeek-V4-Flash: 284.33B total parameters with 13.17B active per token. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

llama-modchip is a set of ggml/llama.cpp kernel patches, quantization tools and runtime utilities for running large sparse Mixture-of-Experts models on an ordinary CPU. Its headline target is DeepSeek-V4-Flash: 284.33B total parameters with 13.17B active per token. The project requantizes that checkpoint from 76.87 GiB to 56.56 GiB (1.71 bits per weight) in its own "Fractal-64" block format, so the model plus a 256K-token KV cache fits in the RAM of a 64 GB machine without swap. The README reports a peak resident size of 57.61 GiB at 256K context. Smaller MoE models such as NVIDIA Nemotron-3.5-30B-A3B run in 16–32 GB.

The speed comes from CPU work rather than a GPU. The patches add AVX-512 kernels that keep quantized weights in registers across several tokens at once during prompt reading, a FlashAttention variant for DeepSeek's multi-query attention, and non-temporal loads so streaming MoE weights do not push the KV cache out of the L3 cache. A small n-gram drafter ("Stockfish-PVS", about 1.64 MB) does speculative decoding, and KV-cache snapshots in /dev/shm restore a long agent session in about 13 ms. The project reports 44.36 tok/s prefill and 9.1–14.7 tok/s generation for the 284B model on a 16-core Zen 5 CPU, and 25–111.6 tok/s for the 30B model depending on the CPU. These are the project's own measurements.

The repository was created on 2026-10-29 and is licensed Apache-2.0. Its models use custom tensor types, so stock llama.cpp, Ollama and LM Studio reject them with "unknown tensor type 40"; you run them through the project's ./modchip launcher or its prebuilt llama-server binary. Besides x86-64 with AVX-512 or AVX2, the README covers ARM64 (NEON/SVE), Apple Metal and a hybrid mode that keeps attention on a GPU while the routed experts stay in CPU RAM. The memory margin is tight: the 284B model needs about 57.6–58.9 GiB against roughly 62.5 GiB usable on a 64 GB Linux system.

## What it does

- Fractal-64 (a 2-bit, 64-weight block format) and Fractal-MXFP4 quants that fit 284B DeepSeek-V4-Flash into 56.56 GiB
- AVX-512 register-tiled prefill kernels and an MQA8 FlashAttention path for DeepSeek's attention
- Non-temporal streaming of MoE weights so the KV cache stays in L3
- Stockfish-PVS n-gram drafter for speculative decoding, about 1.64 MB
- KV-cache slot snapshots in /dev/shm for fast restore of long agent sessions
- Hybrid CPU/GPU split that keeps attention and SSM layers on the GPU and routed experts in RAM
- Configurable expert routing: lossless top-6 by default, top-5 or top-4 via LLAMA_MOE_TOP_K for more speed
- One ./modchip launcher with chat, web, bench, eval, quantize and claude (Claude Code against the local engine) commands

## Getting started

The automated path works on Linux and macOS; Windows has its own launcher (windows/AVVIA_MODCHIP.bat, or ./modchip start in Git Bash). The 284B model needs a 64 GB machine; the 30B Nemotron model runs in 16–32 GB.

1. Check your CPU first

   A readiness script checks your hardware without downloading a model.

```bash
curl -sL https://raw.githubusercontent.com/synchopate/llama-modchip/main/scripts/check_cpu_readiness.py | python3
```

2. Clone and start chatting

   The launcher sets things up and opens a terminal chat that shows throughput and drafter acceptance.

```bash
git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat
```

3. Or run the 284B model directly

   Download the Fractal-64 GGUF and the matching prebuilt server from Hugging Face, then start an OpenAI-compatible server on port 8080.

```bash
hf download Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF --local-dir ./dsv4
chmod +x ./dsv4/llama-server-modchip-linux-x86_64
./dsv4/llama-server-modchip-linux-x86_64 -m ./dsv4/DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf -c 32768 -t 16 -fa on --port 8080
```

4. Or the smaller 30B model

   Nemotron-3.5-30B-A3B in Fractal-64 is 14.35 GiB and runs with the full 256K context.

```bash
hf download Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64-CPU-GGUF \
  llama-server-modchip-linux-x86_64 NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf --local-dir .
chmod +x llama-server-modchip-linux-x86_64
./llama-server-modchip-linux-x86_64 -m NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf -c 262144 -t 16 --port 8080
```

## When to use it

- Run a frontier-size open MoE model on a 64 GB workstation or mini PC that has no GPU
- Point Claude Code at a local 256K-context engine with ./modchip claude
- Benchmark your CPU with ./modchip bench and submit the scorecard to the project's leaderboard
- Quantize your own BF16 or Q4_0 MoE GGUFs to Fractal-64 or Fractal-MXFP4 with ./modchip quantize

## Version history

- 2026-11-04 v2.4.0: Faster CPU path for DeepSeek-V4-Flash: new AVX-512 kernels and an MQA8 FlashAttention path raise prefill on a Ryzen 9 9955HX by 77.1% to 44.36 tok/s and lossless decode to 9.06–9.95 tok/s. Also adds a Windows hybrid GPU+CPU pack and AMD Vulkan MXFP4 support.