Accelerate

// Overview

Accelerate is a Hugging Face library that lets you run a standard PyTorch training loop on any hardware setup without rewriting it. You keep your own training code and add only a few lines, and the library handles device placement, distributed processes, and mixed precision behind the scenes.

It is aimed at PyTorch users who want to write their own training loops but do not want to maintain the boilerplate needed for multi-GPU, TPU, or fp16/bf16/fp8 runs. The same script can run on a single CPU for debugging and then on a multi-GPU cluster without changes.

Within efficient training, Accelerate sits between plain PyTorch and full training frameworks. It wraps your model, optimizer, and dataloader through a single Accelerator object, and integrates backends like FSDP and DeepSpeed that you configure rather than hand-code.

// What it does

• Wraps your existing PyTorch model, optimizer, and dataloader with a single Accelerator.prepare() call
• Runs the same script on single CPU, single GPU, multi-GPU, and TPU without code changes
• Supports mixed precision training in fp8, fp16, and bf16
• Optional accelerate config and accelerate launch CLI to set up and start distributed runs
• Integrates DeepSpeed, including a DeepSpeedPlugin to tweak settings from Python
• Supports multi-CPU runs via MPI (Open MPI, Intel MPI, or MVAPICH)

// Getting started

Install the package, add a few lines to your existing PyTorch loop, then optionally configure and launch a distributed run.

pip install accelerate

import torch
import torch.nn.functional as F
from datasets import load_dataset
from accelerate import Accelerator

accelerator = Accelerator()
device = accelerator.device

model = torch.nn.Transformer().to(device)
optimizer = torch.optim.Adam(model.parameters())

dataset = load_dataset('my_dataset')
data = torch.utils.data.DataLoader(dataset, shuffle=True)

model, optimizer, data = accelerator.prepare(model, optimizer, data)

model.train()
for epoch in range(10):
    for source, targets in data:
        source = source.to(device)
        targets = targets.to(device)
        optimizer.zero_grad()
        output = model(source)
        loss = F.cross_entropy(output, targets)
        accelerator.backward(loss)
        optimizer.step()

accelerate config

accelerate launch my_script.py --args_to_my_script

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Take a single-GPU PyTorch training script to multi-GPU without rewriting the training loop
• Debug a model on a local CPU and then run the same code on a TPU or GPU cluster
• Enable mixed precision (fp16/bf16/fp8) training to cut memory use and speed up runs
• Train large models with DeepSpeed or FSDP by configuring backends instead of hand-coding them

// How Accelerate compares

Accelerate alongside other open-source efficient training tools AI/TLDR tracks, ranked by GitHub stars.