ngrok · 2026-08-11 · notable
Annie Sexton — 'Compression is prediction, and LLMs are compressors'
Annie Sexton walks through why a language model and a compression algorithm solve the same problem. Both score how well you can predict the next symbol, and both optimize cross-entropy to do it.
A language model and a zip file are two faces of one idea: guess the next symbol well and you save bits.
What is it?
"Compression is prediction" is an August 11, 2026 post by Annie Sexton on the ngrok blog that argues data compression and language modeling are the same problem wearing different clothes. It is written for engineers who use LLMs daily but have never seen the information-theory link spelled out.
How does it work?
The post starts with entropy coders, which shrink data by giving likely symbols short codes and unlikely symbols long ones — so the better your probability estimate, the smaller the output. Sexton then shows an LLM doing the identical thing: it produces a probability distribution over the next token, and training minimizes cross-entropy, the exact quantity a compressor is trying to lower.
Why does it matter?
Sexton's framing gives a plain answer to what a model is actually learning during pretraining — a probability model of text, scored by how few bits it would need to encode that text. She also names the catch: despite compressing better in theory, an LLM is impractical for everyday compression because the compute and storage cost dwarfs the savings. The post reached the Hacker News front page with 419 points and 168 comments.
Who is it for?
engineers who want the information theory behind LLM training