# TRL

> Post-train and align language models with SFT, DPO, GRPO, and reward modeling

- Category: Training & Fine-Tuning / RLHF & Alignment
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/huggingface/trl
- Website: https://huggingface.co/docs/trl
- Canonical page: https://ai-tldr.dev/tools/trl/
- Updated: 2026-09-10

## In short

TRL (Transformers Reinforcement Learning) is a Hugging Face library for post-training foundation models. It gives you ready-made trainer classes for the common alignment methods: Supervised Fine-Tuning (SFT), reward modeling, Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO). It sits in the RLHF & Alignment category of the Training & Fine-Tuning tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

TRL (Transformers Reinforcement Learning) is a Hugging Face library for post-training foundation models. It gives you ready-made trainer classes for the common alignment methods: Supervised Fine-Tuning (SFT), reward modeling, Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO).

It is built on top of the Transformers ecosystem, so each trainer is a light wrapper around the standard Transformers trainer and works with many model architectures. It is aimed at machine learning engineers and researchers who want to fine-tune or align a model on their own dataset without writing a training loop from scratch.

In the RLHF and alignment space, TRL covers the full pipeline from instruction tuning through preference optimization. It uses Accelerate to scale from a single GPU to multi-node clusters, and integrates with PEFT for LoRA/QLoRA and Unsloth for faster training.

## What it does

- Dedicated trainers for the main post-training methods: SFTTrainer, RewardTrainer, DPOTrainer, and GRPOTrainer
- GRPOTrainer implements Group Relative Policy Optimization, which is more memory-efficient than PPO
- Scales from a single GPU to multi-node clusters via Accelerate, with support for DDP, DeepSpeed ZeRO, and FSDP
- PEFT integration enables training large models on modest hardware through quantization and LoRA/QLoRA
- Unsloth integration uses optimized kernels to speed up training
- Command line interface lets you run SFT or DPO without writing code

## Getting started

Install TRL with pip, then pick a trainer for your method. The examples below come straight from the project README.

1. Install the package

   Install TRL from PyPI.

```bash
pip install trl
```

2. Run supervised fine-tuning

   Use SFTTrainer to fine-tune a model on a dataset. Pass a model id and a training dataset, then call train().

```python
from trl import SFTTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,
)
trainer.train()
```

3. Align with preferences using DPO

   DPOTrainer runs Direct Preference Optimization on a preference dataset.

```python
from datasets import load_dataset
from trl import DPOTrainer

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,
)
trainer.train()
```

4. Or skip the code with the CLI

   The CLI runs post-training methods like SFT directly from the terminal.

```bash
trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
    --dataset_name trl-lib/Capybara \
    --output_dir Qwen2.5-0.5B-SFT
```

## When to use it

- Instruction-tune a base model on your own dataset with SFTTrainer
- Align a model to human preferences using DPO or a trained reward model
- Apply GRPO to train reasoning or math models in a memory-efficient way
- Fine-tune large models on limited hardware via PEFT LoRA/QLoRA and Unsloth

## Version history

- 2026-09-10 v1.13.0: Documents training at 1,048,576 tokens per step (Qwen3-8B, 380 s/step, 56.2 GB per GPU on one 8xH100 node) and moves the default chunked cross-entropy projection onto tensor cores for up to 1.69x end-to-end throughput. PPOTrainer, PPOConfig and the value-head models are removed; the fused linear DPO/KTO/GRPO/JSD losses are vendored into trl.losses.