# AReaL

> A fully asynchronous RL system for training reasoning and agentic models

- Category: Training & Fine-Tuning / RLHF & Alignment
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/areal-project/AReaL
- Website: https://areal-ai.io
- Canonical page: https://ai-tldr.dev/tools/areal/

## In short

AReaL is a reinforcement learning infrastructure built to connect foundation-model training with agent-based applications. It sits in the RLHF & Alignment category of the Training & Fine-Tuning tools and is open-source, written in Python and licensed Apache-2.0. To get started: clone and install.

## Overview

AReaL is a reinforcement learning infrastructure built to connect foundation-model training with agent-based applications. It was developed by researchers and engineers from Tsinghua IIIS and the AReaL Team at Ant Group, and is built around a fully asynchronous RL training paradigm rather than the synchronous generate-then-train loop most RL stacks use.

Asynchrony is the point. In a synchronous system every GPU waits for the slowest rollout in the batch, which is painful for multi-turn agents whose episode lengths vary wildly. AReaL decouples generation from training so both run continuously; the project's v0.3 release reported a 2.77x speedup over synchronous systems with comparable or better training performance, and notes that the design also simplifies multi-turn agentic RL setup considerably.

AReaL 2.0, released July 2026, refactored the system into a microservice architecture with independent training, inference, agent and weight-update services, and shipped the Hermes online RL loop plus end-to-end SWE RL training examples. A distinguishing feature is online RL for black-box agent applications: you point an existing agent at AReaL's RL service by replacing its base_url and api_key, so agents written against OpenClaw, the OpenAI Agents SDK or CAMEL-AI can be trained without code changes.

## What it does

- Fully asynchronous RL training that decouples generation from updates — the project reports a 2.77x speedup over synchronous systems at comparable quality
- Microservice architecture (AReaL 2.0) with independent training, inference, agent and weight-update services
- Online RL for black-box agents by swapping base_url and api_key — no changes to the agent's own code
- AReaL-lite: an algorithm-first API with roughly 80% fewer lines of code that keeps about 90% of the full system's performance
- Worked examples across math, coding, search, tool-integrated reasoning and customer service, including Tau2-Bench and SWE agent training
- SGLang by default with an optional vLLM path, plus a maintained branch for Ascend NPU devices

## Getting started

AReaL is installed from source with uv. The training scripts download the sample dataset (openai/gsm8k) and model (Qwen/Qwen2-1.5B-Instruct) for you, so a single-node run needs no extra setup.

1. Clone and install

   Install the matching prebuilt flash-attn wheel first to avoid compiling it from source, then sync the CUDA extra. SGLang is the default inference backend; a vLLM variant of the lockfile is included.

```bash
git clone https://github.com/areal-project/AReaL
cd AReaL
pip install uv
# Install a flash-attn pre-built wheel matching your Python version first
uv sync --extra cuda
```

2. Run a single-node GRPO training job

   The math example trains on GSM8K with the local scheduler; the dataset and base model are fetched automatically.

```bash
python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local
```

3. Scale out on a Ray cluster

   Point the config at shared storage and switch the scheduler to ray to train across nodes.

```bash
python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml \
  cluster.n_nodes=2 cluster.n_gpus_per_node=8 \
  cluster.fileroot=/path/to/nfs \
  scheduler.type=ray
```

4. Train an existing agent with online RL

   For black-box agent applications, keep the agent as-is and point it at AReaL's RL service by replacing base_url and api_key. The repository ships worked examples for OpenClaw, the OpenAI Agents SDK, CAMEL-AI and SWE agents; see the agentic RL tutorial in the docs.

## When to use it

- Train a multi-turn agent with RL where episode lengths vary widely and a synchronous batch would leave GPUs idle
- Post-train a reasoning model on math or code with GRPO, PPO, DAPO, RLOO or GSPO from the bundled recipes
- Apply online RL to an agent you already run in production by repointing its base_url, without rewriting it for a training framework
- Reproduce or extend published agentic RL results — search agents, SWE agents and Tau2-Bench customer service are all worked examples