point.free · 2026-06-01 · notable
A 10-Year-Old Xeon Is All You Need — Christina Sørensen Runs Gemma 4 26B-A4B MoE on a 2016 E5-2620 v4 With No GPU at ~12 Tokens/sec
Christina Sørensen (cafkafk) gets Gemma 4's 26B Mixture-of-Experts checkpoint generating at reading speed on a decade-old Xeon E5-2620 v4 with 128GB DDR3 RAM and no GPU, leaning on ik_llama.cpp, MTP speculative decoding, CPU Flash Attention, MLA compression, and runtime expert-routing tuning.
Modern frontier MoE inference, on a decade-old server CPU, at human reading speed.
Key specs
| Context window | 262K |
|---|---|
| Hardware | Intel Xeon E5-2620 v4 (2016), 128GB DDR3 |
| Model | Gemma 4 26B-A4B MoE |
| Throughput | ~11.94 tokens/sec |
| Total memory | 82GB (25GB weights + 56GB KV cache) |
| Hn score | 672 points / 272 comments |
What is it?
A long-form engineering write-up from Christina Sørensen — NixOS Steering Committee member and eza maintainer — describing what it actually took to run Google's Gemma 4 26B-A4B Mixture-of-Experts model on a single 2016-era Intel Xeon E5-2620 v4 with 128GB of DDR3, no GPU, and a 262K context window. The blog post hit #3 on Hacker News with 672 points and 272 comments on June 1.
How does it work?
Sørensen uses ik_llama.cpp (a fork tuned for CPU inference) plus a stack of optimizations: MTP-drafter-based speculative decoding to reduce memory-bandwidth pressure on the verifier, a CPU-side Flash Attention implementation that avoids materializing N×N matrices, Multi-Head Latent Attention to compress the KV cache, runtime weight repacking aligned to L2/L3 cache lines, and explicit expert-routing tuning to keep the MoE from thrashing the cache. Total footprint lands at ~82GB (25GB weights + 56GB KV at 262K context) and the system generates at ~11.94 tokens/sec.
Why does it matter?
It's a real data point for the 'frontier MoE inference can run on commodity CPUs' thesis. Most local-LLM commentary assumes consumer GPUs are the floor; this post documents the exact knobs and trade-offs that turn a recycled 2016 server into a usable Gemma 4 endpoint, and the comment thread has practitioners trading numbers from their own rigs.
Who is it for?
Local-LLM tinkerers, CPU-inference engineers, and anyone running models on recycled hardware.
Try it
Read the post on point.free/blog/gemma-4-on-a-2016-xeon