rLLM

// Overview

rLLM is an open-source framework for training language agents with reinforcement learning. Its organising idea is that the agent you evaluate and the agent you train should be the same code: you write a rollout function, rLLM captures the LLM calls it makes, and those captured traces become the training signal.

That is possible because rLLM inserts a model gateway between your agent and the model. During training, the agent's base URL points at the gateway, which routes requests by session and transparently captures token IDs and logprobs. Your agent code never changes, which is why rLLM can drive off-the-shelf CLI harnesses — Claude Code, Codex, Terminus-2, mini-swe-agent, opencode and others — as well as agents wrapped from LangGraph, the OpenAI Agents SDK or a plain OpenAI client.

The rest of the framework is deliberately swappable. Rollouts run in Docker, Daytona, Modal or locally, with snapshot and warm-pool acceleration to keep training-scale rollouts affordable. Training backends — verl for distributed multi-GPU, tinker for single-machine, fireworks for the Fireworks platform — switch with a single flag, and GRPO, REINFORCE, RLOO, SFT and on-policy distillation are all supported. The project reports state-of-the-art open-source results including DeepScaleR-1.5B, DeepCoder-14B, DeepSWE-32B and FinQA-4B.

// What it does

• Any harness: 10+ CLI harnesses (Claude Code, Codex, Terminus-2, mini-swe-agent, opencode) plus Harbor-compatible task dirs, or wrap your own with @rllm.rollout
• Any sandbox: Docker, Daytona, Modal or local, with snapshot and warm-pool acceleration for cheap rollouts at scale
• Multiple training backends behind one API — verl (distributed multi-GPU), tinker (single-machine) and fireworks — switched with one flag
• 60+ integrated benchmarks including Terminal-Bench 2.0, SWE-bench, SkillsBench, AIME, MATH-500 and GPQA, auto-pulled by `rllm eval <name>`
• Multiple training methods: GRPO, REINFORCE, RLOO, SFT and on-policy distillation
• A model gateway that captures token IDs and logprobs automatically, so the same agent code serves both eval and training

// Getting started

rLLM requires Python 3.11 or newer. The base install targets the tinker backend for single-machine training; distributed and hosted backends are extras. You can drive everything from the CLI or from the Python API.

uv pip install "rllm @ git+https://github.com/rllm-org/rllm.git"

# Distributed multi-GPU training (verl + vLLM/SGLang)
uv pip install "rllm[verl] @ git+https://github.com/rllm-org/rllm.git"

# 1. Configure your model provider
rllm model setup

# 2. Evaluate on a benchmark
rllm eval gsm8k

# 3. Train with RL
rllm train gsm8k

from openai import OpenAI
import rllm
from rllm.types import AgentConfig, Episode, Task, Trajectory

@rllm.rollout
def solve(task: Task, config: AgentConfig) -> Episode:
    client = OpenAI(base_url=config.base_url, api_key="EMPTY")
    response = client.chat.completions.create(
        model=config.model,
        messages=[{"role": "user", "content": task.instruction}],
    )
    answer = response.choices[0].message.content or ""
    return Episode(
        trajectories=[Trajectory(name="solver", steps=[])],
        artifacts={"answer": answer},
    )

from rllm.trainer import AgentTrainer

trainer = AgentTrainer(
    backend="tinker",
    agent_flow=solve,
    evaluator=score,
    config=config,
    train_dataset=dataset,
)
trainer.train()

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Post-train a coding model by running a real CLI harness such as Claude Code or opencode as the rollout and rewarding the tests it makes pass
• Evaluate an agent on 60+ built-in benchmarks first, then train on the same task definition without rewriting the agent
• Switch from single-machine experimentation on tinker to distributed multi-GPU training on verl with one flag as a recipe matures
• Apply GRPO or on-policy distillation to an existing LangGraph or OpenAI Agents SDK agent without porting it to a new framework

// How rLLM compares

rLLM alongside other open-source rlhf & alignment tools AI/TLDR tracks, ranked by GitHub stars.