# verl

> Reinforcement learning post-training for LLMs, from GRPO and PPO to large MoE models

- Category: Training & Fine-Tuning / RLHF & Alignment
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/verl-project/verl
- Website: https://verl.readthedocs.io
- Canonical page: https://ai-tldr.dev/tools/verl/

## In short

verl is a reinforcement learning training library for large language models, started by the ByteDance Seed team and maintained by the verl community. It is the open-source version of the HybridFlow paper, and it focuses on the RL post-training stage that comes after pretraining and supervised fine-tuning. It sits in the RLHF & Alignment category of the Training & Fine-Tuning tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

verl is a reinforcement learning training library for large language models, started by the ByteDance Seed team and maintained by the verl community. It is the open-source version of the HybridFlow paper, and it focuses on the RL post-training stage that comes after pretraining and supervised fine-tuning.

It is aimed at researchers and ML engineers who want to run RL algorithms such as GRPO and PPO on top of existing model infrastructure. verl decouples the computation and data parts of an RL pipeline, so it can plug into training backends like FSDP and Megatron-LM and inference engines like vLLM and SGLang, and it works with HuggingFace models.

Within the RLHF and alignment category, verl handles the training loop itself: generating rollouts, scoring them, and updating the policy. It supports flexible mapping of models onto GPUs, which lets the same code scale from a single 24 GB GPU demo to large clusters training MoE models.

## What it does

- Build GRPO, PPO, and other RL dataflows with the hybrid-controller programming model in a few lines of code
- Integrates with existing training backends (FSDP, Megatron-LM) and inference engines (vLLM, SGLang) through modular APIs
- Flexible device mapping places models on different sets of GPUs for better resource use across cluster sizes
- 3D-HybridEngine reshards the actor model to cut memory redundancy and communication when switching between training and generation
- Ready integration with HuggingFace models
- Scales from a single-GPU example up to large MoE models such as DeepSeek-671B and Qwen3-235B with the Megatron backend

## Getting started

Install verl from source, preprocess a dataset, then launch a PPO training run. A GPU with at least 24 GB of memory is recommended for the demo; Python >= 3.10 and CUDA >= 12.8 are required.

1. Install from source

   Clone the repository and install it in editable mode. Add the vllm or sglang extra to pull in an inference engine.

```bash
git clone https://github.com/verl-project/verl.git
cd verl
pip install --no-deps -e .
pip install -e ".[vllm]"
```

2. Prepare the dataset

   Preprocess the GSM8K math dataset into the parquet files verl expects.

```bash
python3 examples/data_preprocess/gsm8k.py --local_save_dir ~/data/gsm8k
```

3. Run a PPO training job

   Launch the PPO trainer on a small Qwen model. This is the minimal single-GPU demo from the docs; tune the config keys for your hardware.

```bash
PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  data.val_files=$HOME/data/gsm8k/test.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.rollout.name=vllm \
  critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  trainer.n_gpus_per_node=1 \
  trainer.nnodes=1 \
  trainer.total_epochs=15 2>&1 | tee verl_demo.log
```

## When to use it

- Post-train an instruction-tuned model with GRPO or PPO to improve reasoning on math and coding tasks
- Reproduce or build on published RL recipes such as DAPO and ReTool
- Scale RL training of large MoE models across multi-GPU clusters using the Megatron backend
- Experiment with custom reward functions and RL dataflows while reusing existing FSDP, vLLM, or SGLang infrastructure