# OpenRLHF

> Scalable RLHF training for large language models on Ray and vLLM

- Category: Training & Fine-Tuning / RLHF & Alignment
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/OpenRLHF/OpenRLHF
- Website: https://openrlhf.readthedocs.io
- Canonical page: https://ai-tldr.dev/tools/openrlhf/

## In short

OpenRLHF is an open-source framework for reinforcement learning from human feedback (RLHF), the process of aligning a language model with human preferences after pretraining. It is built on Ray for distributed scheduling and vLLM for fast generation, with DeepSpeed handling training. It sits in the RLHF & Alignment category of the Training & Fine-Tuning tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

OpenRLHF is an open-source framework for reinforcement learning from human feedback (RLHF), the process of aligning a language model with human preferences after pretraining. It is built on Ray for distributed scheduling and vLLM for fast generation, with DeepSpeed handling training. This split lets it place the actor, reward, reference, and critic models on separate GPUs and scale to models with 70B+ parameters.

It is aimed at ML engineers and researchers who want to run the full RLHF pipeline—supervised fine-tuning, reward modeling, and reinforcement learning—without building the distributed plumbing themselves. It ships command-line entry points and example scripts for common setups, so you can start from a working recipe and adjust it.

Within the RLHF and alignment category, OpenRLHF focuses on the training infrastructure rather than on datasets or evaluation. It supports several RL algorithms (PPO, GRPO, REINFORCE++, RLOO) and includes a hybrid engine mode that lets models and vLLM engines share GPU resources to reduce idle time on limited hardware.

## What it does

- Built on a Ray + vLLM + DeepSpeed distributed stack that separates actor, reward, reference, and critic models across GPUs
- Supports multiple RL algorithms: PPO, GRPO, REINFORCE++, and RLOO
- Scales RLHF training to models with 70B+ parameters
- Hybrid engine scheduling lets models and vLLM engines share GPUs to cut idle time
- Covers the full pipeline: supervised fine-tuning, reward modeling, and RL training
- Async RLHF and agent-based RLHF via --train.async_enable and --train.agent_func_path, plus optional LoRA

## Getting started

Install OpenRLHF with pip (vLLM is an optional extra), then launch one of its CLI training entry points such as supervised fine-tuning.

1. Install with pip

   Install the base package, or add the vLLM extra for generation acceleration used during RL training.

```bash
pip install openrlhf            # Basic
pip install openrlhf[vllm]      # + vLLM
```

2. Or install from source

   Clone the repository and install in editable mode if you want to modify the code or run the example scripts.

```bash
git clone https://github.com/OpenRLHF/OpenRLHF.git
cd OpenRLHF
pip install -e .
```

3. Run supervised fine-tuning

   Launch the SFT entry point with DeepSpeed. This example fine-tunes Llama 3 8B on the OpenOrca dataset.

```bash
deepspeed --module openrlhf.cli.train_sft \
   --data.max_len 4096 \
   --data.dataset Open-Orca/OpenOrca \
   --data.input_key question \
   --data.output_key response \
   --train.batch_size 256 \
   --train.micro_batch_size 2 \
   --actor.model_name_or_path meta-llama/Meta-Llama-3-8B \
   --ckpt.output_dir ./checkpoint/llama3-8b-sft \
   --ds.zero_stage 2 \
   --train.max_epochs 1 \
   --adam.lr 5e-6
```

## When to use it

- Aligning an open-weight LLM with human preferences using PPO, GRPO, or REINFORCE++
- Running RLHF training on large models (up to 70B+) across multiple GPUs without writing the distributed scheduling yourself
- Reproducing reasoning-model RL recipes (e.g. DeepSeek-R1-style training) from the provided example scripts
- Training reward models and running supervised fine-tuning as the first stages of an alignment pipeline