Sebastian Raschka · 2026-07-28 · notable
Sebastian Raschka — Kimi K3's NoPE, LatentMoE, and attention residuals
Sebastian Raschka reads the Kimi K3 architecture end to end: a scaled-up Kimi Linear with LatentMoE compression, NoPE across every layer, and attention residuals that cost ~4% training compute for a real loss win.
Raschka turns Kimi K3's dense architecture diagram into a plain-English walk through — component by component.
What is it?
Kimi K3 Architecture Notes is a July 28 post by Sebastian Raschka that reads Moonshot's 2.8T open-weight release layer by layer. Raschka frames K3 as a production-scale Kimi Linear, ~58× larger than the earlier 48B model.
How does it work?
The write-up highlights three novel choices: LatentMoE, which compresses the giant MoE linear layers similar to multi-head latent attention; NoPE (no positional embeddings) across every layer, the first frontier-level model to drop RoPE entirely; and cross-layer attention residuals that connect residual streams with attention weights, adding ~4% training and ~2% inference cost for a validation-loss win.
Why does it matter?
This is the clearest independent read of K3 anyone has published. Anyone deciding whether to fine-tune or self-host Kimi K3 now has a walk-through of what is new versus prior Kimi Linear or Nemotron 3 Ultra, and why the trade-offs might land in their workload.
Who is it for?
ML researchers, model implementers, and infra engineers evaluating Kimi K3
Try it
Read: sebastianraschka.com/blog/2026/kimi-k3-architecture-notes.html