Overview
benchmax is a Python SDK for defining and training reinforcement-learning environments: tools, rewards and datasets as code. Instead of hand-assembling a post-training pipeline, you describe the environment a model should learn in — what it can call, what counts as success, what data it sees — and hand that object to a trainer.
Two base classes cover the common shapes. Extending BaseEnv gives you a basic model-and-tool loop; HarborEnv is for training against an arbitrary harness and sandbox, so the environment can be a real program the model drives rather than a synthetic task. The companion castform package layers on dataset-generation helpers, environment validation and job launching, and its setup command signs you in and creates a starter environment.
The SDK is the open, Apache-2.0 half of Castform, a commercial platform that runs the reinforcement-learning post-training itself so developers do not manage GPUs. It requires Python 3.12.
What it does
- Define an RL environment in code: tools, rewards and datasets in one object
- BaseEnv for a straightforward model-and-tool loop
- HarborEnv for training against an arbitrary harness and sandbox
- Dataset-generation helpers and environment validation before you spend a training run
- Launch training jobs on the Castform platform from the CLI
- Apache-2.0 SDK, installable with uv or pip on Python 3.12
Getting started
Install the castform package (which brings the benchmax SDK), sign in, and start from the generated starter environment.
Install
Either installer works; the CLI is called castform.
uv tool install -U castform
# or: pip install -U castform
castform --versionSign in and scaffold an environment
setup signs you in and creates a starter environment to edit.
castform setupWrite the environment
Subclass BaseEnv for a model-and-tool loop, or HarborEnv when the model should drive a real harness inside a sandbox. The repository's examples show both shapes.
# extend BaseEnv for a tool loop, or HarborEnv for a harness + sandboxValidate, then launch
The castform package adds dataset-generation helpers, environment validation and training-job launching, so a broken reward is caught before a run starts.
# validate the environment, generate data, then launch the training jobCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Post-train a small open-weight model on a narrow task instead of paying for a frontier model on every request
- Express a reward as code — what a good trajectory looks like in your own product
- Train an agent against the actual harness and sandbox it will run in, not a stand-in
- Run reinforcement learning without provisioning or babysitting GPUs
How benchmax compares
benchmax alongside other open-source rlhf & alignment tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Open-R1 | ★ 26.5k | An open reproduction of the DeepSeek-R1 reasoning pipeline, with scripts for GRPO training and reasoning-data generation. |
| verl | ★ 23.4k | Volcano Engine's RL post-training framework (HybridFlow) for building GRPO, PPO, and other RL pipelines on top of FSDP, Megatron, and vLLM. |
| TRL | ★ 19.2k | Hugging Face's post-training library with trainers for SFT, reward modeling, DPO, PPO, and GRPO to align language models with preferences. |
| Agent Lightning | ★ 18k | An open-source trainer from Microsoft that improves AI agents built with any framework using reinforcement learning, prompt optimization, and supervised fine-tuning. |
| ART | ★ 10.7k | OpenPipe's Agent Reinforcement Trainer for post-training LLM agents on multi-step tasks using GRPO and rule- or judge-based rewards. |
| OpenRLHF | ★ 10k | A Ray- and vLLM-based RLHF framework that scales PPO, GRPO, and REINFORCE++ training to models with 70B+ parameters. |
| slime | ★ 8.4k | THUDM's SGLang-native RL post-training framework, pairing Megatron training with SGLang rollout in a single dataflow; the RL framework behind the GLM model releases. |
| benchmax | ★ 51 | A Python SDK for defining reinforcement-learning environments as code — tools, rewards and datasets — and launching training jobs on them |