DeepSpeed

// Overview

DeepSpeed is a deep learning optimization library for PyTorch. Its core idea, ZeRO (Zero Redundancy Optimizer), splits model states, gradients, and optimizer state across the GPUs in a job instead of copying them to every device. With ZeRO-Infinity and offloading, it can also move data to CPU memory or NVMe, so you can train models that would not fit in GPU memory alone.

It is built for teams training large models that do not fit on a single GPU, or that train too slowly with plain data parallelism. DeepSpeed has been used to train large language models such as BLOOM (176B) and Megatron-Turing NLG (530B). You wrap a normal PyTorch model and run it through DeepSpeed's engine and launcher.

In the efficient-training space, DeepSpeed sits alongside the model code rather than replacing it. It works with plain PyTorch and integrates with higher-level tools like Hugging Face Accelerate and PyTorch Lightning, so you can adopt ZeRO and offloading without rewriting your training loop from scratch.

// What it does

• ZeRO partitions optimizer state, gradients, and parameters across GPUs to cut per-device memory use
• ZeRO-Infinity and offloading move data to CPU or NVMe so larger models fit in available memory
• 3D parallelism combines data, tensor, and pipeline parallelism for large-scale training
• A JSON config file (ds_config.json) controls optimizer, precision, and ZeRO settings without code changes
• The deepspeed launcher handles single-node and multi-node distributed runs via a hostfile
• Integrates with Hugging Face Accelerate and PyTorch Lightning

// Getting started

Install the package, wrap your PyTorch model with deepspeed.initialize, and launch training with the deepspeed CLI and a config file.

pip install deepspeed

model_engine, optimizer, _, _ = deepspeed.initialize(args=cmd_args,
                                                     model=model,
                                                     model_parameters=params)

for step, batch in enumerate(data_loader):
    # forward() method
    loss = model_engine(batch)

    # runs backpropagation
    model_engine.backward(loss)

    # weight update
    model_engine.step()

deepspeed --hostfile=myhostfile <client_entry.py> <client args> \
  --deepspeed --deepspeed_config ds_config.json

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Training a large language model that does not fit on a single GPU by partitioning its state across many GPUs with ZeRO
• Fitting a bigger model into limited GPU memory by offloading optimizer state and parameters to CPU or NVMe
• Scaling an existing PyTorch training job to multiple nodes using the deepspeed launcher and a hostfile
• Adding ZeRO and offloading to a Hugging Face Accelerate or PyTorch Lightning workflow without rewriting the training loop

// How DeepSpeed compares

DeepSpeed alongside other open-source efficient training tools AI/TLDR tracks, ranked by GitHub stars.