Open-AutoGLM

// Overview

Open-AutoGLM is a mobile agent framework from Z.ai (the lab behind the GLM models). You describe a task in plain language — "Open eBay and search for wireless earphones" — and the framework takes a screenshot of the phone, has a vision-language model work out what is on screen, plans the next action, and performs it on the device.

It belongs in the computer-use category because it automates a real device rather than calling APIs. Control happens through ADB (Android Debug Bridge) for Android, HDC for HarmonyOS, and WebDriverAgent for iPhone, so the agent taps, types, and swipes exactly as a person would. It ships a sensitive-operation confirmation step and lets you take over by hand when a login screen or verification code appears.

The models behind it are published separately: AutoGLM-Phone-9B is tuned for Chinese apps and AutoGLM-Phone-9B-Multilingual covers English and other languages. You can serve either yourself with vLLM or SGLang, or point the framework at a hosted endpoint such as Z.ai, Novita AI, or Parasail. The repository is Apache-2.0 and the project states it is for research and learning only.

// What it does

• Vision-language screen understanding — the agent reads the live interface instead of relying on accessibility trees or hard-coded selectors
• Cross-platform device control via ADB (Android), HDC (HarmonyOS), and WebDriverAgent (iOS)
• Natural-language task input, with an interactive mode or a one-shot task passed on the command line
• Sensitive-operation confirmation plus manual takeover for logins and verification codes
• Remote debugging over WiFi, so the phone does not need to stay tethered by USB
• Runs against a self-hosted vLLM/SGLang endpoint or a third-party host, and integrates with the Midscene.js UI-automation SDK

// Getting started

You need Python 3.10+, a device with developer mode and USB debugging enabled, and either your own model server or an API key from a provider that hosts the AutoGLM phone models.

pip install -r requirements.txt
pip install -e .

# Android
adb devices

# HarmonyOS
hdc list targets

python3 -m vllm.entrypoints.openai.api_server \
 --served-model-name autoglm-phone-9b-multilingual \
 --allowed-local-media-path / \
 --mm-encoder-tp-mode data \
 --mm_processor_cache_type shm \
 --mm_processor_kwargs "{\"max_pixels\":5000000}" \
 --max-model-len 25480 \
 --chat-template-content-format string \
 --limit-mm-per-prompt "{\"image\":10}" \
 --model zai-org/AutoGLM-Phone-9B-Multilingual \
 --port 8000

python scripts/check_deployment_en.py --base-url http://localhost:8000/v1 --model autoglm-phone-9b-multilingual

# Self-hosted
python main.py --base-url http://localhost:8000/v1 "Open Maps and search for nearby coffee shops"

# Hosted on Z.ai
python main.py --base-url https://api.z.ai/api/paas/v4 --model "autoglm-phone-multilingual" --apikey "your-z-ai-api-key" "Open Chrome browser"

from phone_agent import PhoneAgent
from phone_agent.model import ModelConfig

model_config = ModelConfig(
    base_url="http://localhost:8000/v1",
    model_name="autoglm-phone-9b-multilingual",
)

agent = PhoneAgent(model_config=model_config)
result = agent.run("Open eBay and search for wireless earphones")
print(result)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Automate multi-step errands inside mobile apps that expose no API — searching a marketplace, filling a form, working through a settings flow
• Build and test mobile agent research on real hardware rather than a simulator
• Run repeatable QA passes across Android and HarmonyOS builds from one natural-language script
• Control a phone bench remotely over WiFi from a development machine

// How Open-AutoGLM compares

Open-AutoGLM alongside other open-source computer & browser use tools AI/TLDR tracks, ranked by GitHub stars.