Stratechery · 2026-05-11 · notable
Ben Thompson: The Inference Shift — Why Agentic Inference Will Favor Memory Over Speed
Ben Thompson splits AI inference into two regimes: 'answer inference' that needs low latency for humans, and 'agentic inference' where memory hierarchy and cost matter more than raw speed.
An essay arguing the next phase of AI compute splits in two: speed-bound 'answer inference' for humans, capacity-bound 'agentic inference' for everything else.
Key specs
| Wse3 sram bandwidth | 21 PB/s |
|---|---|
| H100 hbm bandwidth | 3.35 TB/s |
| Bandwidth delta | ~6,000x |
| Colossus gpus | 220,000+ |
| Colossus power mw | 300 |
What is it?
A Monday Stratechery essay by Ben Thompson, timed against Cerebras's bumped IPO range and the new Anthropic-SpaceX Colossus 1 deal. Thompson reframes the AI compute conversation from one number (FLOPs) into two distinct markets: answer inference (humans are watching, latency is the bottleneck) and agentic inference (no human in the loop, batch and memory capacity dominate).
How does it work?
Thompson contrasts Cerebras's WSE-3 wafer — 44GB of on-die SRAM at 21 PB/s — against an H100's 80GB of HBM at 3.35 TB/s, calling the wafer-scale design a roughly 6,000x bandwidth advantage that makes sense only when speed-of-token-out is the binding constraint. For agentic workloads with long context, parallel tool calls, and no human waiting, he argues the better hardware target is the opposite end: lots of slower, cheaper memory and older silicon nodes, used at high duty cycle. The Anthropic-SpaceX Colossus 1 lease — 220,000+ NVIDIA GPUs across 300 MW — is offered as the early shape of an agentic-inference fleet.
Why does it matter?
The piece is the cleanest articulation yet of why NVIDIA's HBM-stacked premium chips are not automatically the answer for the next leg of compute demand. If Thompson is right, public-market valuations for Cerebras, Groq, and the rest of the 'inference-only' silicon vendors get pulled in two directions at once: Cerebras prices its WSE for the answer-inference niche, while agentic fleets compound on memory-rich, older nodes that look more like Anthropic's NVIDIA-Colossus deal.
Who is it for?
infra investors, AI hardware analysts, agent-platform builders
Try it
Read the public Stratechery post — Monday updates are free this week