RoboLab

// Overview

RoboLab is a task-based evaluation benchmark for robot manipulation policies, built on NVIDIA Isaac Lab. It ships 100+ manipulation tasks with automated success detection, a server-client policy architecture and multi-environment parallel evaluation, aimed at reproducible large-scale benchmarking of generalist robot policies in simulation rather than at one-off demos.

The headline task set, RoboLab-120, is 120 new benchmark tasks spanning pick-and-place, stacking, rearrangement and tool use. Each carries a language instruction and automated success or failure detection built from composable predicates, so scoring does not depend on a human watching replays. Tasks are not tied to a specific robot embodiment, so any robot compatible with Isaac Lab can be plugged in, and the repository ships asset libraries of objects, scenes and curated backgrounds for building new tasks of your own.

Around the tasks sit the parts that make a benchmark usable: your model runs as a standalone server and RoboLab connects to it through a lightweight inference client, episodes run in parallel across vectorized environments with per-environment termination, and a self-contained web dashboard browses scenes and tasks, replays episode videos and compares results across experiments. The project also ships `/robolab-scenegen` and `/robolab-taskgen` Claude Code skills for generating new scenes and tasks from natural language, and an ecosystem page listing projects built on top of it — including RoboVoLo, a long-horizon and reasoning-heavy task library released in August 2026.

// What it does

• RoboLab-120: 120 manipulation tasks with language instructions and automated success detection via composable predicates
• Embodiment-agnostic — plug in any robot that works with Isaac Lab
• Server-client policy architecture: your model runs as its own server, RoboLab connects via a thin inference client
• Multi-environment parallel evaluation with vectorized conditionals and per-environment termination
• Asset libraries of objects, scenes and backgrounds for authoring new tasks
• Self-contained results dashboard with episode video replay and cross-experiment comparison
• Scene and task generation from natural language through the bundled robolab-scenegen and robolab-taskgen skills

// Getting started

RoboLab needs uv, a system ffmpeg for video recording, and an IsaacSim/IsaacLab stack chosen at install time through a mutually-exclusive extra. The two stacks cannot coexist in one environment, so install each into its own venv if you need both.

sudo apt install ffmpeg
git clone <repo_url>
cd robolab
uv venv --python 3.11
source .venv/bin/activate
uv sync --extra isaac50

uv run pytest tests/

python examples/run_empty.py --headless
python examples/run_recorded.py --headless
python examples/run_gripper_toggle.py --task BananaInBowlTask --headless

cd robolab
uv run python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask --num-envs 10

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Benchmark a generalist manipulation policy against a fixed 120-task suite with automated, reproducible scoring
• Compare several policy checkpoints across experiments and replay the episodes where they failed
• Build a custom evaluation suite for your own robot using the bundled object, scene and background libraries
• Run large evaluation sweeps in parallel across vectorized Isaac Lab environments instead of one episode at a time

// How RoboLab compares

RoboLab alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.