# Sebastian Raschka — Kimi K3's NoPE, LatentMoE, and attention residuals

> Raschka turns Kimi K3's dense architecture diagram into a plain-English walk through — component by component.

- Org: Sebastian Raschka
- Date: 2026-07-28
- Category: article
- Source: https://sebastianraschka.com/blog/2026/kimi-k3-architecture-notes.html
- Canonical page: https://ai-tldr.dev/releases/sebastian-raschka-kimi-k3-arch-jul28/

## In short

Sebastian Raschka reads the Kimi K3 architecture end to end: a scaled-up Kimi Linear with LatentMoE compression, NoPE across every layer, and attention residuals that cost ~4% training compute for a real loss win.

## What it is

Kimi K3 Architecture Notes is a July 28 post by Sebastian Raschka that reads Moonshot's 2.8T open-weight release layer by layer. Raschka frames K3 as a production-scale Kimi Linear, ~58× larger than the earlier 48B model.

## How it works

The write-up highlights three novel choices: LatentMoE, which compresses the giant MoE linear layers similar to multi-head latent attention; NoPE (no positional embeddings) across every layer, the first frontier-level model to drop RoPE entirely; and cross-layer attention residuals that connect residual streams with attention weights, adding ~4% training and ~2% inference cost for a validation-loss win.

## Why it matters

This is the clearest independent read of K3 anyone has published. Anyone deciding whether to fine-tune or self-host Kimi K3 now has a walk-through of what is new versus prior Kimi Linear or Nemotron 3 Ultra, and why the trade-offs might land in their workload.

## Who it's for

ML researchers, model implementers, and infra engineers evaluating Kimi K3

## Try it

Read: sebastianraschka.com/blog/2026/kimi-k3-architecture-notes.html

## Links

- [Full Article](https://sebastianraschka.com/blog/2026/kimi-k3-architecture-notes.html)
- [Kimi K3 on Hugging Face](https://huggingface.co/moonshotai/Kimi-K3)
- [Author on X](https://x.com/rasbt)