Overview
AReaL is a reinforcement learning infrastructure built to connect foundation-model training with agent-based applications. It was developed by researchers and engineers from Tsinghua IIIS and the AReaL Team at Ant Group, and is built around a fully asynchronous RL training paradigm rather than the synchronous generate-then-train loop most RL stacks use.
Asynchrony is the point. In a synchronous system every GPU waits for the slowest rollout in the batch, which is painful for multi-turn agents whose episode lengths vary wildly. AReaL decouples generation from training so both run continuously; the project's v0.3 release reported a 2.77x speedup over synchronous systems with comparable or better training performance, and notes that the design also simplifies multi-turn agentic RL setup considerably.
AReaL 2.0, released July 2026, refactored the system into a microservice architecture with independent training, inference, agent and weight-update services, and shipped the Hermes online RL loop plus end-to-end SWE RL training examples. A distinguishing feature is online RL for black-box agent applications: you point an existing agent at AReaL's RL service by replacing its base_url and api_key, so agents written against OpenClaw, the OpenAI Agents SDK or CAMEL-AI can be trained without code changes.
What it does
- Fully asynchronous RL training that decouples generation from updates — the project reports a 2.77x speedup over synchronous systems at comparable quality
- Microservice architecture (AReaL 2.0) with independent training, inference, agent and weight-update services
- Online RL for black-box agents by swapping base_url and api_key — no changes to the agent's own code
- AReaL-lite: an algorithm-first API with roughly 80% fewer lines of code that keeps about 90% of the full system's performance
- Worked examples across math, coding, search, tool-integrated reasoning and customer service, including Tau2-Bench and SWE agent training
- SGLang by default with an optional vLLM path, plus a maintained branch for Ascend NPU devices
Getting started
AReaL is installed from source with uv. The training scripts download the sample dataset (openai/gsm8k) and model (Qwen/Qwen2-1.5B-Instruct) for you, so a single-node run needs no extra setup.
Clone and install
Install the matching prebuilt flash-attn wheel first to avoid compiling it from source, then sync the CUDA extra. SGLang is the default inference backend; a vLLM variant of the lockfile is included.
git clone https://github.com/areal-project/AReaL
cd AReaL
pip install uv
# Install a flash-attn pre-built wheel matching your Python version first
uv sync --extra cudaRun a single-node GRPO training job
The math example trains on GSM8K with the local scheduler; the dataset and base model are fetched automatically.
python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=localScale out on a Ray cluster
Point the config at shared storage and switch the scheduler to ray to train across nodes.
python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml \
cluster.n_nodes=2 cluster.n_gpus_per_node=8 \
cluster.fileroot=/path/to/nfs \
scheduler.type=rayTrain an existing agent with online RL
For black-box agent applications, keep the agent as-is and point it at AReaL's RL service by replacing base_url and api_key. The repository ships worked examples for OpenClaw, the OpenAI Agents SDK, CAMEL-AI and SWE agents; see the agentic RL tutorial in the docs.
Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Train a multi-turn agent with RL where episode lengths vary widely and a synchronous batch would leave GPUs idle
- Post-train a reasoning model on math or code with GRPO, PPO, DAPO, RLOO or GSPO from the bundled recipes
- Apply online RL to an agent you already run in production by repointing its base_url, without rewriting it for a training framework
- Reproduce or extend published agentic RL results — search agents, SWE agents and Tau2-Bench customer service are all worked examples
How AReaL compares
AReaL alongside other open-source rlhf & alignment tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Heretic | ★ 31.7k | A command-line tool that removes refusal behaviour from transformer language models by directional ablation, using an Optuna parameter search that co-minimizes refusals and KL divergence from the original model. |
| Open-R1 | ★ 26.5k | An open reproduction of the DeepSeek-R1 reasoning pipeline, with scripts for GRPO training and reasoning-data generation. |
| verl | ★ 23.5k | Volcano Engine's RL post-training framework (HybridFlow) for building GRPO, PPO, and other RL pipelines on top of FSDP, Megatron, and vLLM. |
| TRL | ★ 19.3k | Hugging Face's post-training library with trainers for SFT, reward modeling, DPO, PPO, and GRPO to align language models with preferences. |
| Agent Lightning | ★ 18.3k | An open-source trainer from Microsoft that improves AI agents built with any framework using reinforcement learning, prompt optimization, and supervised fine-tuning. |
| ART | ★ 10.7k | OpenPipe's Agent Reinforcement Trainer for post-training LLM agents on multi-step tasks using GRPO and rule- or judge-based rewards. |
| OpenRLHF | ★ 10k | A Ray- and vLLM-based RLHF framework that scales PPO, GRPO, and REINFORCE++ training to models with 70B+ parameters. |
| AReaL | ★ 5.8k | A fully asynchronous RL system for training reasoning and agentic models |