Overview
Megatron-LM is NVIDIA's repository for training large transformer models across many GPUs. It ships two parts: Megatron-LM, a reference example with pre-configured training scripts, and Megatron Core, a composable library of GPU-optimized building blocks you can drop into your own training framework.
Megatron Core provides transformer building blocks, several parallelism strategies (tensor, pipeline, data, expert, and context parallelism), mixed-precision support (FP16, BF16, FP8, and FP4), and ready model architectures. It is aimed at framework developers and ML engineers building custom training pipelines, while the Megatron-LM scripts suit research teams learning distributed training or running quick experiments.
As an efficient-training tool, it targets the case where a model or batch no longer fits on a single GPU. It splits the work across devices so you can pretrain and post-train large language and Mixture-of-Experts models on multi-GPU clusters.
What it does
- Multiple parallelism strategies: tensor (TP), pipeline (PP), data (DP), expert (EP), and context (CP) parallelism
- Mixed-precision training with FP16, BF16, FP8, and FP4
- Megatron Core: composable, GPU-optimized transformer building blocks for custom frameworks
- Pre-configured training scripts and ready-to-use examples in Megatron-LM
- Post-training support including quantization, distillation, pruning, and RLHF
- Megatron Bridge for bidirectional Hugging Face to Megatron checkpoint conversion
Getting started
Install Megatron Core from PyPI, or clone the repository to get the reference training scripts and examples.
Install from PyPI
Install the Megatron Core library directly into your environment.
uv pip install megatron-coreOr install from source
Clone the repository and install in editable mode to also get the training scripts and examples. Building can use a lot of memory; limit parallel jobs with MAX_JOBS if the build runs out of memory.
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
uv pip install -e .Next steps
Follow NVIDIA's quickstart for an end-to-end training run with data preparation, and the parallelism guide to scale across GPUs. NGC container setup is covered in the Installation Guide.
Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Pretraining a large language model that does not fit on a single GPU, split across a multi-GPU cluster
- Training Mixture-of-Experts models such as DeepSeek, Mixtral, or Qwen3 with expert parallelism
- Building a custom training framework on top of Megatron Core's transformer and parallelism building blocks
- Post-training tasks like quantization, distillation, pruning, or RLHF on large models
How Megatron-LM compares
Megatron-LM alongside other open-source efficient training tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| DeepSpeed | ★ 42.8k | A deep learning optimization library whose ZeRO memory partitioning and offloading let you train very large models across many GPUs. |
| Megatron-LM | ★ 17.1k | GPU-optimized library for training transformer models at scale |
| Accelerate | ★ 9.8k | A library that runs the same PyTorch training code across CPUs, multiple GPUs, and TPUs while handling mixed precision, FSDP, and DeepSpeed. |
| Liger-Kernel | ★ 6.5k | A set of fused Triton kernels for common LLM layers that raises training throughput and lowers memory use as a drop-in replacement. |
| TorchTitan | ★ 5.5k | A PyTorch-native platform for pre-training large models that combines FSDP, tensor, pipeline, and context parallelism in one codebase. |
| Nanotron | ★ 2.8k | Hugging Face's minimal library for pre-training LLMs with 3D parallelism, designed to be readable and easy to modify. |