# ClawGUI

> Train, evaluate and deploy GUI agents on real phones from one framework

- Category: Agents & Automation / Computer & Browser Use
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/ZJU-REAL/ClawGUI
- Website: https://zju-real.github.io/ClawGUI-Page/
- Canonical page: https://ai-tldr.dev/tools/clawgui/
- Updated: 2026-04-13

## In short

ClawGUI is a research framework that covers the whole lifecycle of a GUI agent — online reinforcement-learning training, standardized evaluation, and deployment onto real devices. It sits in the Computer & Browser Use category of the Agents & Automation tools and is open-source, written in Python and licensed Apache-2.0. To get started: clone the repository.

## Overview

ClawGUI is a research framework that covers the whole lifecycle of a GUI agent — online reinforcement-learning training, standardized evaluation, and deployment onto real devices. Its premise is that these three problems are tightly coupled but rarely solved together: you need an environment to train the agent online, rigorous benchmarks to measure what it learned, and a production path to run it on a real phone.

The repository is five independent modules, each with its own environment and README. ClawGUI-RL trains agents online across dozens of Docker-based Android emulators in parallel or directly on physical and cloud devices, replacing standard GRPO with GiGPO plus a Process Reward Model for step-level rewards, with spare-server rotation for failover and trajectory replay. ClawGUI-Eval is a three-stage Infer → Judge → Metric pipeline covering six grounding benchmarks — ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G and AndroidControl — and 11+ models, with a reported 95.8% reproduction rate against official results, so numbers from different papers become comparable.

The remaining modules handle deployment. ClawGUI-Agent, built on OpenClaw and nanobot, drives Android over ADB, HarmonyOS over HDC and iOS over XCTest from 12+ chat platforms, and can kick off the full evaluation pipeline from a single sentence. ClawGUI-Skills implements the training-free skill-evolution architecture from the project's paper, storing procedural knowledge as structured packages that agents retrieve, inject, diagnose and revise, with four modes (`off`, `trace`, `reuse`, `evolve`) and disabled by default to avoid context cost. ClawGUI-APP runs the brain and GUI agent entirely on one Android phone using Shizuku for high-privilege control, with no desktop coordinator. ClawGUI-2B, a 2B agent trained end to end with this pipeline, is published as the validation case.

## What it does

- Online RL training across dozens of parallel Docker Android environments or on physical and cloud devices
- GiGPO plus a Process Reward Model for fine-grained step-level rewards instead of standard GRPO
- Evaluation across six GUI grounding benchmarks and 11+ models with a reported 95.8% reproduction rate
- Local GPU (transformers) or remote OpenAI-compatible API backends, with multi-GPU, multi-thread inference and automatic resume
- Real-device deployment on Android (ADB), HarmonyOS (HDC) and iOS (XCTest) from 12+ chat platforms
- Training-free self-evolving skills: structured packages with retrieval, failure diagnosis and restricted revision
- Phone-only mode via Shizuku that runs the brain and the GUI agent on a single Android device

## Getting started

ClawGUI is a monorepo of independent modules; each has its own environment and installation instructions. Clone it first, then work in the module you need — RL for training, Eval for measurement, Agent or APP for deployment. Python 3.12 is the target.

1. Clone the repository

   Everything hangs off the top-level clone; each module lives in its own directory.

```bash
git clone https://github.com/ZJU-REAL/ClawGUI.git
cd ClawGUI
```

2. Pick a module

   clawgui-rl/ trains agents online with RL. clawgui-eval/ measures them across six benchmarks. clawgui-agent/ deploys them onto real devices. clawgui-skills/ adds self-evolving skill packages. clawgui-app/ runs the whole stack on one Android phone. Follow the README in the directory you chose for its full install steps.

3. Evaluate an existing model first

   ClawGUI-Eval is the cheapest way to see the framework work: run its Infer → Judge → Metric pipeline against a model you already have, using either a local GPU backend through transformers or a remote OpenAI-compatible endpoint. Multi-GPU runs resume automatically if interrupted.

4. Drive a real device

   ClawGUI-Agent connects to an Android device over ADB, HarmonyOS over HDC or iOS over XCTest, and exposes a Gradio web UI for device management, task execution and memory inspection. It can also trigger the whole benchmark pipeline in one sentence — for example, asking it to benchmark a model on ScreenSpot-Pro.

## When to use it

- Train a GUI agent with online reinforcement learning against real Android apps instead of static screenshots
- Reproduce and compare published GUI-grounding results across models with one measurement pipeline
- Ship a phone-controlling agent that users talk to from an existing chat platform
- Run an agent entirely on-device, with no desktop coordinator, using Shizuku for privileged control

## Version history

- 2026-04-13: ClawGUI released as a unified framework for GUI agents — ClawGUI-RL for online RL training, ClawGUI-Eval for benchmarking and ClawGUI-Agent for deployment — with ClawGUI-2B published as its end-to-end validation.