Overview
RAGEN (Reasoning AGENt) is a reinforcement-learning framework for training LLM agents over multiple turns, from the MLL Lab at Northwestern University. Single-turn RLHF optimises one response; RAGEN optimises a whole trajectory — the agent reasons, acts on an environment, observes the result, and the reward lands on the sequence rather than on any one reply. The interactions are formulated as an MDP whose states and actions are token sequences, so the model can reason over environment dynamics rather than pattern-match a single prompt.
The algorithm underneath is StarPO — State-Thinking-Actions-Reward Policy Optimization. In the rollout stage the model generates several trajectories from an initial state, producing a reasoning-guided action at each step in the form `<think>…</think><ans> action </ans>`; the environment returns a reward and the next state. In the update stage StarPO optimises whole trajectories using importance sampling, supporting both PPO (token-level advantages via a value function) and GRPO (a normalised reward assigned to the full trajectory), and both trajectory-level and turn-wise training.
What sets RAGEN apart from a generic RL harness is that it is built to *diagnose* agent RL rather than just run it. The v1 work named the Echo Trap; RAGEN-2, released March 2026, studies reasoning collapse — specifically *template collapse*, where reasoning looks varied within one input but becomes input-agnostic across inputs, a failure entropy alone cannot see. RAGEN-2 decomposes reasoning quality onto two axes, within-input diversity (conditional entropy H(Z|X)) and cross-input distinguishability (mutual information I(X;Z)), yielding four reasoning regimes, and adds SNR-Adaptive Filtering: a lightweight rollout filter that uses reward variance as a proxy to select high-signal prompts each iteration.
Ten environments ship in the box — Sokoban, FrozenLake, WebShop, DeepCoder, SearchQA, Lean, Bandit, Countdown, MetaMathQA and Sudoku — behind a gym-compatible interface, so adding your own is a matter of implementing the same surface. veRL is integrated as a submodule, and the codebase is split into an Environment State Manager, a Context Manager and an Agent Proxy. Both papers are on arXiv (2504.20073 for v1, 2604.06268 for v2).
What it does
- StarPO: trajectory-level policy optimisation for multi-turn agents, with PPO and GRPO update rules and both trajectory-level and turn-wise training
- Ten built-in environments (Sokoban, FrozenLake, WebShop, DeepCoder, SearchQA, Lean, Bandit, Countdown, MetaMathQA, Sudoku) behind a gym-compatible interface
- Reasoning-collapse diagnostics: conditional entropy and mutual-information proxies that expose template collapse, which entropy alone cannot detect
- SNR-Adaptive Filtering — reward-variance-based rollout filtering (top-p, top-k, linear) that the authors report improves training across algorithms, model scales and modalities
- A modular codebase — Environment State Manager, Context Manager, Agent Proxy — with veRL integrated as a submodule
- A one-command setup script, config-name-driven training runs, and published experiment logs on Weights & Biases
Getting started
RAGEN is cloned and set up with its own script into a conda environment; training runs are selected by config name.
Clone and set up the environment
The setup script pulls the veRL submodule and installs dependencies. Add --with-search to include the search environment; WebShop has its own guide under docs/experiment_webshop_release.md.
git clone https://github.com/mll-lab-nu/RAGEN.git
cd RAGEN
conda create -n ragen python=3.12 -y && conda activate ragen
bash scripts/setup_ragen.shTrain on Sokoban with the defaults
Each environment has a config. The default run uses no rollout filtering.
python train.py --config-name _2_sokobanTurn on SNR-Adaptive Filtering
RAGEN-2's rollout filter is switched on with two overrides. Other strategies (top-k, linear mode) are covered in docs/guide_rollout_filtering.md.
python train.py --config-name _2_sokoban \
actor_rollout_ref.rollout_filter_strategy=top_p \
actor_rollout_ref.rollout.rollout_filter_value=0.9
Evaluate a trained agent
The agent proxy runs evaluation against the same config.
python -m ragen.llm_agent.agent_proxy --config-name _2_sokobanCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Reach for it when an agent has to be trained across many turns of tool use, not scored on a single response
- Reach for it when RL runs are unstable and you need to tell entropy collapse apart from template collapse before changing hyperparameters
- Reach for it as a testbed: ten environments behind one gym-compatible interface make it straightforward to check whether a result holds beyond one task
- Reach for it to reproduce or extend the StarPO and SNR-Adaptive Filtering experiments — both papers, the configs and the W&B logs are published
How RAGEN compares
RAGEN alongside other open-source rlhf & alignment tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Heretic | ★ 36.3k | A command-line tool that removes refusal behaviour from transformer language models by directional ablation, using an Optuna parameter search that co-minimizes refusals and KL divergence from the original model. |
| Open-R1 | ★ 26.5k | An open reproduction of the DeepSeek-R1 reasoning pipeline, with scripts for GRPO training and reasoning-data generation. |
| verl | ★ 24k | Volcano Engine's RL post-training framework (HybridFlow) for building GRPO, PPO, and other RL pipelines on top of FSDP, Megatron, and vLLM. |
| TRL | ★ 19.6k | Hugging Face's post-training library with trainers for SFT, reward modeling, DPO, PPO, and GRPO to align language models with preferences. |
| Agent Lightning | ★ 18.8k | An open-source trainer from Microsoft that improves AI agents built with any framework using reinforcement learning, prompt optimization, and supervised fine-tuning. |
| ART | ★ 10.8k | OpenPipe's Agent Reinforcement Trainer for post-training LLM agents on multi-step tasks using GRPO and rule- or judge-based rewards. |
| OpenRLHF | ★ 10.1k | A Ray- and vLLM-based RLHF framework that scales PPO, GRPO, and REINFORCE++ training to models with 70B+ parameters. |
| RAGEN | ★ 2.8k | Multi-turn RL for LLM agents built on StarPO, with ten built-in environments and diagnostics that name the failure modes — the Echo Trap and reasoning collapse — instead of leaving you to guess |