# rLLM

> Train language agents with RL on any harness, sandbox and training backend

- Category: Training & Fine-Tuning / RLHF & Alignment
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/rllm-org/rllm
- Website: https://rllm-project.com
- Canonical page: https://ai-tldr.dev/tools/rllm/

## In short

rLLM is an open-source framework for training language agents with reinforcement learning. It sits in the RLHF & Alignment category of the Training & Fine-Tuning tools and is open-source, written in Python and licensed Apache-2.0. To get started: install rLLM.

## Overview

rLLM is an open-source framework for training language agents with reinforcement learning. Its organising idea is that the agent you evaluate and the agent you train should be the same code: you write a rollout function, rLLM captures the LLM calls it makes, and those captured traces become the training signal.

That is possible because rLLM inserts a model gateway between your agent and the model. During training, the agent's base URL points at the gateway, which routes requests by session and transparently captures token IDs and logprobs. Your agent code never changes, which is why rLLM can drive off-the-shelf CLI harnesses — Claude Code, Codex, Terminus-2, mini-swe-agent, opencode and others — as well as agents wrapped from LangGraph, the OpenAI Agents SDK or a plain OpenAI client.

The rest of the framework is deliberately swappable. Rollouts run in Docker, Daytona, Modal or locally, with snapshot and warm-pool acceleration to keep training-scale rollouts affordable. Training backends — verl for distributed multi-GPU, tinker for single-machine, fireworks for the Fireworks platform — switch with a single flag, and GRPO, REINFORCE, RLOO, SFT and on-policy distillation are all supported. The project reports state-of-the-art open-source results including DeepScaleR-1.5B, DeepCoder-14B, DeepSWE-32B and FinQA-4B.

## What it does

- Any harness: 10+ CLI harnesses (Claude Code, Codex, Terminus-2, mini-swe-agent, opencode) plus Harbor-compatible task dirs, or wrap your own with @rllm.rollout
- Any sandbox: Docker, Daytona, Modal or local, with snapshot and warm-pool acceleration for cheap rollouts at scale
- Multiple training backends behind one API — verl (distributed multi-GPU), tinker (single-machine) and fireworks — switched with one flag
- 60+ integrated benchmarks including Terminal-Bench 2.0, SWE-bench, SkillsBench, AIME, MATH-500 and GPQA, auto-pulled by `rllm eval <name>`
- Multiple training methods: GRPO, REINFORCE, RLOO, SFT and on-policy distillation
- A model gateway that captures token IDs and logprobs automatically, so the same agent code serves both eval and training

## Getting started

rLLM requires Python 3.11 or newer. The base install targets the tinker backend for single-machine training; distributed and hosted backends are extras. You can drive everything from the CLI or from the Python API.

1. Install rLLM

   Install from the repository with uv. Add the verl extra for distributed multi-GPU training, or fireworks for the Fireworks platform.

```bash
uv pip install "rllm @ git+https://github.com/rllm-org/rllm.git"

# Distributed multi-GPU training (verl + vLLM/SGLang)
uv pip install "rllm[verl] @ git+https://github.com/rllm-org/rllm.git"
```

2. Evaluate and train from the CLI

   No code needed for the built-in benchmarks — configure a provider, evaluate, then train on the same task.

```bash
# 1. Configure your model provider
rllm model setup

# 2. Evaluate on a benchmark
rllm eval gsm8k

# 3. Train with RL
rllm train gsm8k
```

3. Wrap your own agent with the Python API

   Decorate a rollout function with @rllm.rollout. During training, config.base_url points at rLLM's gateway, so the same function is used for eval and for collecting training traces.

```python
from openai import OpenAI
import rllm
from rllm.types import AgentConfig, Episode, Task, Trajectory

@rllm.rollout
def solve(task: Task, config: AgentConfig) -> Episode:
    client = OpenAI(base_url=config.base_url, api_key="EMPTY")
    response = client.chat.completions.create(
        model=config.model,
        messages=[{"role": "user", "content": task.instruction}],
    )
    answer = response.choices[0].message.content or ""
    return Episode(
        trajectories=[Trajectory(name="solver", steps=[])],
        artifacts={"answer": answer},
    )
```

4. Score the result and hand it to the trainer

   An @rllm.evaluator function turns an episode into a reward; AgentTrainer runs the RL loop against the chosen backend.

```python
from rllm.trainer import AgentTrainer

trainer = AgentTrainer(
    backend="tinker",
    agent_flow=solve,
    evaluator=score,
    config=config,
    train_dataset=dataset,
)
trainer.train()
```

## When to use it

- Post-train a coding model by running a real CLI harness such as Claude Code or opencode as the rollout and rewarding the tests it makes pass
- Evaluate an agent on 60+ built-in benchmarks first, then train on the same task definition without rewriting the agent
- Switch from single-machine experimentation on tinker to distributed multi-GPU training on verl with one flag as a recipe matures
- Apply GRPO or on-policy distillation to an existing LangGraph or OpenAI Agents SDK agent without porting it to a new framework