ClawGUI

// Overview

ClawGUI is a research framework that covers the whole lifecycle of a GUI agent — online reinforcement-learning training, standardized evaluation, and deployment onto real devices. Its premise is that these three problems are tightly coupled but rarely solved together: you need an environment to train the agent online, rigorous benchmarks to measure what it learned, and a production path to run it on a real phone.

The repository is five independent modules, each with its own environment and README. ClawGUI-RL trains agents online across dozens of Docker-based Android emulators in parallel or directly on physical and cloud devices, replacing standard GRPO with GiGPO plus a Process Reward Model for step-level rewards, with spare-server rotation for failover and trajectory replay. ClawGUI-Eval is a three-stage Infer → Judge → Metric pipeline covering six grounding benchmarks — ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G and AndroidControl — and 11+ models, with a reported 95.8% reproduction rate against official results, so numbers from different papers become comparable.

The remaining modules handle deployment. ClawGUI-Agent, built on OpenClaw and nanobot, drives Android over ADB, HarmonyOS over HDC and iOS over XCTest from 12+ chat platforms, and can kick off the full evaluation pipeline from a single sentence. ClawGUI-Skills implements the training-free skill-evolution architecture from the project's paper, storing procedural knowledge as structured packages that agents retrieve, inject, diagnose and revise, with four modes (`off`, `trace`, `reuse`, `evolve`) and disabled by default to avoid context cost. ClawGUI-APP runs the brain and GUI agent entirely on one Android phone using Shizuku for high-privilege control, with no desktop coordinator. ClawGUI-2B, a 2B agent trained end to end with this pipeline, is published as the validation case.

// What it does

• Online RL training across dozens of parallel Docker Android environments or on physical and cloud devices
• GiGPO plus a Process Reward Model for fine-grained step-level rewards instead of standard GRPO
• Evaluation across six GUI grounding benchmarks and 11+ models with a reported 95.8% reproduction rate
• Local GPU (transformers) or remote OpenAI-compatible API backends, with multi-GPU, multi-thread inference and automatic resume
• Real-device deployment on Android (ADB), HarmonyOS (HDC) and iOS (XCTest) from 12+ chat platforms
• Training-free self-evolving skills: structured packages with retrieval, failure diagnosis and restricted revision
• Phone-only mode via Shizuku that runs the brain and the GUI agent on a single Android device

// Getting started

ClawGUI is a monorepo of independent modules; each has its own environment and installation instructions. Clone it first, then work in the module you need — RL for training, Eval for measurement, Agent or APP for deployment. Python 3.12 is the target.

git clone https://github.com/ZJU-REAL/ClawGUI.git
cd ClawGUI

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Train a GUI agent with online reinforcement learning against real Android apps instead of static screenshots
• Reproduce and compare published GUI-grounding results across models with one measurement pipeline
• Ship a phone-controlling agent that users talk to from an existing chat platform
• Run an agent entirely on-device, with no desktop coordinator, using Shizuku for privileged control

// How ClawGUI compares

ClawGUI alongside other open-source computer & browser use tools AI/TLDR tracks, ranked by GitHub stars.