# Open-AutoGLM

> A phone agent that reads your screen and taps through apps for you

- Category: Agents & Automation / Computer & Browser Use
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/zai-org/Open-AutoGLM
- Canonical page: https://ai-tldr.dev/tools/open-autoglm/

## In short

Open-AutoGLM is a mobile agent framework from Z.ai (the lab behind the GLM models). It sits in the Computer & Browser Use category of the Agents & Automation tools and is open-source, written in Python and licensed Apache-2.0. To get started: install the framework.

## Overview

Open-AutoGLM is a mobile agent framework from Z.ai (the lab behind the GLM models). You describe a task in plain language — "Open eBay and search for wireless earphones" — and the framework takes a screenshot of the phone, has a vision-language model work out what is on screen, plans the next action, and performs it on the device.

It belongs in the computer-use category because it automates a real device rather than calling APIs. Control happens through ADB (Android Debug Bridge) for Android, HDC for HarmonyOS, and WebDriverAgent for iPhone, so the agent taps, types, and swipes exactly as a person would. It ships a sensitive-operation confirmation step and lets you take over by hand when a login screen or verification code appears.

The models behind it are published separately: AutoGLM-Phone-9B is tuned for Chinese apps and AutoGLM-Phone-9B-Multilingual covers English and other languages. You can serve either yourself with vLLM or SGLang, or point the framework at a hosted endpoint such as Z.ai, Novita AI, or Parasail. The repository is Apache-2.0 and the project states it is for research and learning only.

## What it does

- Vision-language screen understanding — the agent reads the live interface instead of relying on accessibility trees or hard-coded selectors
- Cross-platform device control via ADB (Android), HDC (HarmonyOS), and WebDriverAgent (iOS)
- Natural-language task input, with an interactive mode or a one-shot task passed on the command line
- Sensitive-operation confirmation plus manual takeover for logins and verification codes
- Remote debugging over WiFi, so the phone does not need to stay tethered by USB
- Runs against a self-hosted vLLM/SGLang endpoint or a third-party host, and integrates with the Midscene.js UI-automation SDK

## Getting started

You need Python 3.10+, a device with developer mode and USB debugging enabled, and either your own model server or an API key from a provider that hosts the AutoGLM phone models.

1. Install the framework

   Clone the repository, then install the dependencies and the package itself.

```bash
pip install -r requirements.txt
pip install -e .
```

2. Connect and check the device

   Enable developer mode and USB debugging on the phone, connect it with a data-capable USB cable, and confirm the tooling can see it. Android devices also need the ADB Keyboard app installed for text input.

```bash
# Android
adb devices

# HarmonyOS
hdc list targets
```

3. Point it at a model

   Either serve the open weights yourself with vLLM, or use a hosted endpoint. The README's vLLM invocation must be followed closely — the startup flags matter.

```bash
python3 -m vllm.entrypoints.openai.api_server \
 --served-model-name autoglm-phone-9b-multilingual \
 --allowed-local-media-path / \
 --mm-encoder-tp-mode data \
 --mm_processor_cache_type shm \
 --mm_processor_kwargs "{\"max_pixels\":5000000}" \
 --max-model-len 25480 \
 --chat-template-content-format string \
 --limit-mm-per-prompt "{\"image\":10}" \
 --model zai-org/AutoGLM-Phone-9B-Multilingual \
 --port 8000
```

4. Verify the model service

   A helper script confirms the endpoint answers and reports token statistics before you let the agent touch the phone.

```bash
python scripts/check_deployment_en.py --base-url http://localhost:8000/v1 --model autoglm-phone-9b-multilingual
```

5. Run a task

   Pass a task as an argument for a one-shot run, or omit it for interactive mode. Use --device-type hdc for HarmonyOS devices.

```bash
# Self-hosted
python main.py --base-url http://localhost:8000/v1 "Open Maps and search for nearby coffee shops"

# Hosted on Z.ai
python main.py --base-url https://api.z.ai/api/paas/v4 --model "autoglm-phone-multilingual" --apikey "your-z-ai-api-key" "Open Chrome browser"
```

6. Or drive it from Python

   The PhoneAgent class exposes the same loop for embedding in your own code.

```python
from phone_agent import PhoneAgent
from phone_agent.model import ModelConfig

model_config = ModelConfig(
    base_url="http://localhost:8000/v1",
    model_name="autoglm-phone-9b-multilingual",
)

agent = PhoneAgent(model_config=model_config)
result = agent.run("Open eBay and search for wireless earphones")
print(result)
```

## When to use it

- Automate multi-step errands inside mobile apps that expose no API — searching a marketplace, filling a form, working through a settings flow
- Build and test mobile agent research on real hardware rather than a simulator
- Run repeatable QA passes across Android and HarmonyOS builds from one natural-language script
- Control a phone bench remotely over WiFi from a development machine