# Decoupled DiLoCo: 236× Less Bandwidth for Distributed LLM Training

> Google DeepMind cuts inter-datacenter training bandwidth 236× while maintaining 88% goodput when chips fail — validated at 12B scale.

- Org: Google DeepMind
- Date: 2026-04-23
- Category: algorithm, paper
- Source: https://deepmind.google/blog/decoupled-diloco/
- Canonical page: https://ai-tldr.dev/releases/google-decoupled-diloco/

## In short

Google DeepMind's Decoupled DiLoCo cuts cross-datacenter training bandwidth 236× (198 Gbps → 0.84 Gbps) while hitting 88% goodput under hardware failures vs 27% for standard data-parallel. Validated on a 12B model across four US regions.

## What it is

Decoupled DiLoCo is a distributed LLM pre-training method from Google DeepMind that separates local gradient updates within each datacenter from periodic cross-datacenter synchronization. Where conventional data-parallel training requires constant 198 Gbps cross-datacenter links, Decoupled DiLoCo communicates asynchronously with only 0.84 Gbps, using periodic parameter averaging ('DiLoCo steps') instead of per-step gradient synchronization.

## How it works

Each compute island runs local SGD steps independently, then periodically exchanges averaged model parameters via an inner-outer loop. These cross-datacenter averaging steps are fully decoupled: they don't block local training when a remote site is slow or failing. When a hardware failure takes out a slice of TPUs, the rest of the cluster continues at full speed — achieving 88% goodput vs 27% with standard data-parallel under simulated high-failure conditions with 1.2 million chips. Validated on a 12B parameter model across four US regions over multi-day runs.

## Why it matters

Training frontier models currently requires co-located supercomputers because cross-datacenter bandwidth is too expensive for synchronous training. Decoupled DiLoCo makes geographically distributed training viable at 1/236th the bandwidth cost, opening the door to pooling compute across multiple data centers and tolerating chip failures without cluster-wide downtime. The method comes with a production-scale validation, not just a proof-of-concept.

## Who it's for

ML infrastructure engineers training large models across distributed compute

## Try it

arxiv.org/abs/2604.21428

## Links

- [Blog Post](https://deepmind.google/blog/decoupled-diloco/)
- [Paper](https://arxiv.org/abs/2604.21428)