# RoboLab

> Simulation benchmark and harness for evaluating generalist robot manipulation policies

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/NVlabs/RoboLab
- Website: https://research.nvidia.com/labs/srl/projects/robolab/
- Canonical page: https://ai-tldr.dev/tools/robolab/
- Updated: 2026-04-14

## In short

RoboLab is a task-based evaluation benchmark for robot manipulation policies, built on NVIDIA Isaac Lab. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0. To get started: install.

## Overview

RoboLab is a task-based evaluation benchmark for robot manipulation policies, built on NVIDIA Isaac Lab. It ships 100+ manipulation tasks with automated success detection, a server-client policy architecture and multi-environment parallel evaluation, aimed at reproducible large-scale benchmarking of generalist robot policies in simulation rather than at one-off demos.

The headline task set, RoboLab-120, is 120 new benchmark tasks spanning pick-and-place, stacking, rearrangement and tool use. Each carries a language instruction and automated success or failure detection built from composable predicates, so scoring does not depend on a human watching replays. Tasks are not tied to a specific robot embodiment, so any robot compatible with Isaac Lab can be plugged in, and the repository ships asset libraries of objects, scenes and curated backgrounds for building new tasks of your own.

Around the tasks sit the parts that make a benchmark usable: your model runs as a standalone server and RoboLab connects to it through a lightweight inference client, episodes run in parallel across vectorized environments with per-environment termination, and a self-contained web dashboard browses scenes and tasks, replays episode videos and compares results across experiments. The project also ships `/robolab-scenegen` and `/robolab-taskgen` Claude Code skills for generating new scenes and tasks from natural language, and an ecosystem page listing projects built on top of it — including RoboVoLo, a long-horizon and reasoning-heavy task library released in August 2026.

## What it does

- RoboLab-120: 120 manipulation tasks with language instructions and automated success detection via composable predicates
- Embodiment-agnostic — plug in any robot that works with Isaac Lab
- Server-client policy architecture: your model runs as its own server, RoboLab connects via a thin inference client
- Multi-environment parallel evaluation with vectorized conditionals and per-environment termination
- Asset libraries of objects, scenes and backgrounds for authoring new tasks
- Self-contained results dashboard with episode video replay and cross-experiment comparison
- Scene and task generation from natural language through the bundled robolab-scenegen and robolab-taskgen skills

## Getting started

RoboLab needs uv, a system ffmpeg for video recording, and an IsaacSim/IsaacLab stack chosen at install time through a mutually-exclusive extra. The two stacks cannot coexist in one environment, so install each into its own venv if you need both.

1. Install

   Clone the repo, create a Python 3.11 environment with uv, and sync with the Isaac extra you want. isaac50 (IsaacSim 5.0 / IsaacLab 2.2.0) is the default; isaac51 targets IsaacSim 5.1 / IsaacLab 2.3.2.post1.

```bash
sudo apt install ffmpeg
git clone <repo_url>
cd robolab
uv venv --python 3.11
source .venv/bin/activate
uv sync --extra isaac50
```

2. Verify the install

   The test suite is the install-verification suite: it checks isaaclab imports, validates every task definition, populates the env factory and runs one full episode. It auto-accepts the NVIDIA Omniverse EULA so the run is fully headless.

```bash
uv run pytest tests/
```

3. Run an episode without a policy

   Sanity-check the simulator before wiring in a model: random actions, recorded-demonstration playback, or a gripper toggle that saves sensor and viewport video under output/.

```bash
python examples/run_empty.py --headless
python examples/run_recorded.py --headless
python examples/run_gripper_toggle.py --task BananaInBowlTask --headless
```

4. Evaluate a policy

   Start your model as a server and let RoboLab drive it. The bundled Pi0.5 backend via OpenPI is the quickest way to see a real evaluation run end to end.

```bash
cd robolab
uv run python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask --num-envs 10
```

5. Inspect the results

   Open the bundled dashboard to browse scenes and tasks, replay episode videos and compare runs across experiments. Output is written to your local folder.

## When to use it

- Benchmark a generalist manipulation policy against a fixed 120-task suite with automated, reproducible scoring
- Compare several policy checkpoints across experiments and replay the episodes where they failed
- Build a custom evaluation suite for your own robot using the bundled object, scene and background libraries
- Run large evaluation sweeps in parallel across vectorized Isaac Lab environments instead of one episode at a time

## Version history

- 2026-04-14: NVIDIA published RoboLab, a 120-task simulation benchmark for robot manipulation policies, alongside its paper and public leaderboard.