Show-Harness

// Overview

Show-Harness is a control layer between a vision-language model and a robot arm, released by Show Lab at the National University of Singapore alongside the paper "Show-Harness: Just a VLM Agent Can Play Robots" (arXiv:2609.10522). Instead of asking the model for joint angles or end-effector poses, it exposes the robot as a compact set of discrete semantic action units — the granularity a language model already reasons about. Embodiment-specific interpreters then ground each unit into motion deterministically, which keeps the VLM directly responsible for every physical decision rather than delegating to a learned low-level controller.

The project runs in two modes through the same interface. In zero-shot mode a closed frontier VLM operates the full plugin harness with no robot-specific training, driven through any OpenAI-compatible endpoint. In fine-tuned mode a small open VLM, trained on demonstrations, emits one action token per step with no planner in front of it — the real-robot configs default to a Qwen3.5-2B adapter. Switching embodiments changes only the interpreter and its `configs/primitives_<embodiment>.yaml`; the model-facing vocabulary and prompt set stay identical across Franka, AgileX Piper (single and dual arm), ManiSkill and Isaac Lab.

Data collection ships with it. GUMI (GUI Manipulation Interface) maps every action unit to a key or button so a human — or a GUI-driving agent — demonstrates a task by playing the robot in a browser, and every step is recorded as a training-ready (observation, action) pair. No teleoperation hardware and no post-processing are involved, and a synthetic tabletop world lets you try the interface before any rig exists. The repository is Apache-2.0; six LoRA adapters and the demonstration corpus are published separately on Hugging Face, and the released adapters were trained against Qwen3.5, Gemma 4 and InternVL3.5 backbones.

// What it does

• Discrete semantic action units as the model-facing interface, grounded into motion by deterministic per-embodiment interpreters
• Two modes on one interface: a frontier VLM zero-shot, or a fine-tuned small VLM emitting one action token per step
• Embodiment-agnostic — Franka, AgileX Piper (single and dual arm), ManiSkill and Isaac Lab share one action vocabulary and one prompt set
• GUMI browser teleoperation: demonstrate a task with keyboard controls, every step recorded as a ready training pair, no teleop hardware
• Nine toggleable harness plugins across the perception, reasoning and action stages, byte-identical to no plugin when disabled
• Provider-agnostic VLM client — talks to any OpenAI-compatible endpoint, or serve a local model with the bundled vLLM script
• Self-contained LoRA fine-tuning pipeline built on LLaMA-Factory, plus six released adapters and the demonstration corpus on Hugging Face

// Getting started

The harness and the local serving stack have conflicting pins, so they live in separate virtual environments. Start with `base` — it is enough to collect demonstrations in simulation and to drive a robot from a hosted model. Running `bash scripts/setup.sh` with no arguments prints which environments already exist.

git clone https://github.com/showlab/Show-Harness.git
cd Show-Harness
bash scripts/setup.sh base

bash scripts/setup.sh base
.venv/bin/python gumi/collect_rollouts_web.py data/rollouts_demo --sim
# open http://localhost:8600 and drive the gripper with WASD / arrow keys

cp configs/site/franka.yaml.example configs/site/franka.yaml
cp configs/secrets.env.example configs/secrets.env
python scripts/check_setup.py --robot-config configs/robot_franka.yaml
python scripts/run_real.py --robot-config configs/robot_franka.yaml

ADAPTER=qwen3_5_2b WITH_BASE=1 bash scripts/model/download_vlm_model.sh

MODEL=Qwen/Qwen3.5-2B \
  LORA=qwen3_5_2b_showharness_ft=models/Show-Harness-VLMs/qwen3_5_2b \
  FAMILY=qwen3_5 bash scripts/serve_vlm.sh

python scripts/run_real_mvtoken.py --robot-config configs/robot_franka_ft.yaml

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when you want a frontier VLM to control a robot arm without collecting data or training a policy first
• Reach for it when your lab has no teleoperation hardware — GUMI collects demonstrations from a browser
• Reach for it when the same task has to run across several embodiments and you do not want a separate prompt set per robot
• Reach for it when you want to fine-tune a small open VLM into a manipulation policy on a modest GPU budget

// How Show-Harness compares

Show-Harness alongside other open-source robot control stacks tools AI/TLDR tracks, ranked by GitHub stars.