Doubleword · 2026-07-27 · notable
Kimi Delta Attention explained — Doubleword walks the DeltaNet lineage
Doubleword co-founder Jamie Dborin walks step by step through the DeltaNet family of linear-attention variants and shows how you could arrive at Kimi Delta Attention (KDA), the layer now used inside Kimi K3.
A guided walk through the DeltaNet family that lands on Kimi Delta Attention, the linear-attention layer inside Kimi K3.
What is it?
Kimi Delta Attention explained is a technical blog post from Doubleword co-founder Jamie Dborin, published July 27, 2026. It walks the reader through the DeltaNet family of linear-attention variants, one step at a time, until the reader can derive Kimi Delta Attention (KDA) themselves — the exact layer that now runs inside Moonshot's Kimi K3 and recent Qwen models.
How does it work?
The article starts from plain softmax attention, then swaps in linear attention (a fixed-size recurrent state instead of a KV cache), adds the delta rule to make each write correct for what was already stored, adds a gate to control forgetting, and ends at KDA, which sharpens forgetting with channel-wise decay. Each step is motivated by a small edit to the hidden state, so KDA reads as an obvious next move rather than a black box.
Why does it matter?
The DeltaNet lineage is the mechanism behind the linear-attention layers in Kimi K3 and recent Qwen releases, so understanding KDA is the shortest path to knowing why those models keep long contexts fast. Dborin's walkthrough gives practitioners a mental model they can extend when the next linear-attention variant lands.
Who is it for?
ML researchers and engineers working on efficient long-context attention.
Try it
Read the post at blog.doubleword.ai/you-could-have-come-up-with-kimi-delta-attention