█

AI/TLDR

Nanotron

Minimal, readable library for pretraining transformer LLMs with 3D parallelism

github.com/huggingface/nanotron★ 2.8k

Overview

Nanotron is a library from Hugging Face for pretraining transformer models. It gives you a small, readable codebase for training large language models on your own datasets, with the parallelism techniques needed to scale across many GPUs.

It is aimed at researchers and engineers who pretrain models from scratch and want to understand and modify the training code rather than treat it as a black box. The APIs for tensor and pipeline parallelism are explicit, which makes the distributed logic easier to follow and debug.

Within efficient-training tooling, Nanotron focuses on 3D parallelism (data, tensor, and pipeline) plus expert parallelism for Mixture-of-Experts models, so you can fit and train models that do not fit on a single device.

What it does

  • 3D parallelism: data, tensor, and pipeline parallel (DP+TP+PP)
  • Expert parallelism for Mixture-of-Experts (MoE) models
  • AFAB and 1F1B pipeline schedules
  • Explicit TP and PP APIs to make distributed training easier to debug
  • ZeRO-1 optimizer and FP32 gradient accumulation
  • Parameter tying/sharding and worked examples for Llama, Mamba, MoE, DoReMi, and muP

Getting started

Set up a Python environment with PyTorch, install Nanotron, then launch a tiny Llama training run with torchrun. A GPU node (the quick start targets 8 x H100s) is expected.

Create an environment and install PyTorch

Create a Python 3.11 virtual environment with uv, then install PyTorch with the matching CUDA wheel.

bashbash
uv venv nanotron --python 3.11 && source nanotron/bin/activate && uv pip install --upgrade pip
uv pip install torch --index-url https://download.pytorch.org/whl/cu124

Install Nanotron

Install the core package in editable mode from the repo root. For the example scripts, also install the listed extra dependencies.

bashbash
uv pip install -e .
# For the example scripts:
uv pip install datasets transformers datatrove[io] numba wandb

Train a tiny Llama model

Launch training with torchrun using the bundled tiny Llama config. Checkpoints are written to the directory set in the config file.

bashbash
CUDA_DEVICE_MAX_CONNECTIONS=1 torchrun --nproc_per_node=8 run_train.py --config-file examples/config_tiny_llama.yaml

Generate from a checkpoint

Run generation from a saved checkpoint, raising --tp for more GPUs or --pp for very large models.

bashbash
torchrun --nproc_per_node=1 run_generate.py --ckpt-path checkpoints/{checkpoint_number}/ --tp 1 --pp 1

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Pretraining a custom LLM from scratch on your own dataset across multiple GPUs or nodes
  • Scaling a model too large for one device using combined data, tensor, and pipeline parallelism
  • Training Mixture-of-Experts models with expert parallelism
  • Studying or modifying a compact training codebase to experiment with parallelism strategies and schedules

How Nanotron compares

Nanotron alongside other open-source efficient training tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
DeepSpeed★ 42.8kA deep learning optimization library whose ZeRO memory partitioning and offloading let you train very large models across many GPUs.
Megatron-LM★ 17.1kNVIDIA's library for training large transformer models at scale using tensor, pipeline, and sequence parallelism.
Accelerate★ 9.8kA library that runs the same PyTorch training code across CPUs, multiple GPUs, and TPUs while handling mixed precision, FSDP, and DeepSpeed.
Liger-Kernel★ 6.5kA set of fused Triton kernels for common LLM layers that raises training throughput and lowers memory use as a drop-in replacement.
TorchTitan★ 5.5kA PyTorch-native platform for pre-training large models that combines FSDP, tensor, pipeline, and context parallelism in one codebase.
Nanotron★ 2.8kMinimal, readable library for pretraining transformer LLMs with 3D parallelism