Sam Witteveen · 2026-11-09 · notable
Sam Witteveen — 'NVIDIA's Secret 2 Bit Model' on Nemotron 3.5 Bolt 74B-A9B
In short
Sam Witteveen explains how NVIDIA's Nemotron 3.5 Bolt 74B-A9B squeezes into about 24GB with 2-bit experts, covering ParetoQ, pruning and quantization-aware training, then tests it locally on agentic tool calls.
Sam Witteveen · 2026-11-09 · 4 sources checked
A walkthrough of why NVIDIA's 2-bit Nemotron 3.5 Bolt works when most sub-4-bit models break.
What is it?
Sam Witteveen's video breaks down Nemotron 3.5 Bolt 74B-A9B, an NVIDIA model with 74B total and 9B active parameters whose experts are stored at about 2 bits, packing the whole model into roughly 24GB. He uses it to explain why the recipe matters more than the model itself.
How does it work?
The video covers how standard quantization works and why it tends to break below 4 bits, the ParetoQ paper's case for 2-bit models, and mixed precision. It then walks through NVIDIA's three-stage pipeline of pruning, post-training and quantization-aware training, plus NVIDIA's QAD twist.
Why does it matter?
A 74B-class model that fits in 24GB changes what runs on a single workstation GPU. Witteveen demos Bolt on an HP ZGX Fury with an SVG test, agentic tool-calling and concurrency runs, and discusses which hardware it suits.
Who is it for?
developers running models locally
Try it
https://www.youtube.com/watch?v=WJpeW9lCSUM