# Kimi Delta Attention explained — Doubleword walks the DeltaNet lineage

> A guided walk through the DeltaNet family that lands on Kimi Delta Attention, the linear-attention layer inside Kimi K3.

- Org: Doubleword
- Date: 2026-07-27
- Category: article
- Source: https://blog.doubleword.ai/you-could-have-come-up-with-kimi-delta-attention
- Canonical page: https://ai-tldr.dev/releases/doubleword-kimi-delta-attention/

## In short

Doubleword co-founder Jamie Dborin walks step by step through the DeltaNet family of linear-attention variants and shows how you could arrive at Kimi Delta Attention (KDA), the layer now used inside Kimi K3.

## What it is

Kimi Delta Attention explained is a technical blog post from Doubleword co-founder Jamie Dborin, published July 27, 2026. It walks the reader through the DeltaNet family of linear-attention variants, one step at a time, until the reader can derive Kimi Delta Attention (KDA) themselves — the exact layer that now runs inside Moonshot's Kimi K3 and recent Qwen models.

## How it works

The article starts from plain softmax attention, then swaps in linear attention (a fixed-size recurrent state instead of a KV cache), adds the delta rule to make each write correct for what was already stored, adds a gate to control forgetting, and ends at KDA, which sharpens forgetting with channel-wise decay. Each step is motivated by a small edit to the hidden state, so KDA reads as an obvious next move rather than a black box.

## Why it matters

The DeltaNet lineage is the mechanism behind the linear-attention layers in Kimi K3 and recent Qwen releases, so understanding KDA is the shortest path to knowing why those models keep long contexts fast. Dborin's walkthrough gives practitioners a mental model they can extend when the next linear-attention variant lands.

## Who it's for

ML researchers and engineers working on efficient long-context attention.

## Try it

Read the post at blog.doubleword.ai/you-could-have-come-up-with-kimi-delta-attention

## Links

- [Article](https://blog.doubleword.ai/you-could-have-come-up-with-kimi-delta-attention)
- [HN discussion](https://news.ycombinator.com/item?id=49085909)
- [Kimi K3 model](https://huggingface.co/moonshotai/kimi-k3)