# agent-desktop > A computer-use CLI and C library that lets AI agents read and operate desktop apps through the OS accessibility tree - Category: Agents & Automation / Computer & Browser Use - Language: Rust - Licence: Apache-2.0 - Repository: https://github.com/vercel-labs/agent-desktop - Website: https://agent-browser.dev/desktop - Canonical page: https://ai-tldr.dev/tools/agent-desktop/ - Updated: 2026-11-03 ## In short agent-desktop is an open-source tool from Vercel Labs that gives an AI agent hands on a desktop computer. It sits in the Computer & Browser Use category of the Agents & Automation tools and is open-source, written in Rust and licensed Apache-2.0. To get started: install the CLI. ## Overview agent-desktop is an open-source tool from Vercel Labs that gives an AI agent hands on a desktop computer. It is a command-line program and a C library that read the user interface of native apps on macOS, Windows and Linux and act on it: click, type, scroll, manage windows, read notifications and the clipboard. It is a tool that agents call, not an agent itself, and it contains no model. For web pages, the same team points you to its sibling project, agent-browser. The key design choice is the accessibility tree. Instead of matching pixels in screenshots, agent-desktop reads each app through the operating system's own accessibility APIs — the AX API on macOS, UI Automation on Windows and AT-SPI on Linux. A snapshot returns JSON with a snapshot ID and a ref for every element an agent can act on, and actions target those refs (for example `@s8f3k2p9:e8`). The trade-off is stated plainly: an app that publishes no accessibility tree is invisible to it. Every CLI call runs as its own process, and the reasoning loop stays in the agent: observe with a snapshot, act on a ref, then snapshot again to verify. Errors such as `STALE_REF` and `AMBIGUOUS_TARGET` tell the agent to look again. Ref actions are headless by default, so they do not move the pointer or steal focus; a global `--headed` flag switches to physical input. The binary also ships its own skill docs matched to its version, which an agent can load with `agent-desktop skills get desktop`. ## What it does - Reads native desktop apps through the OS accessibility tree on macOS (AX API), Windows (UI Automation) and Linux (AT-SPI) — no pixel matching - JSON snapshots with a snapshot ID and a ref per actionable element; bare refs like `@e8` are rejected so stale targets are caught - About 60 commands with the same names on every platform: observation, ref actions, keyboard and mouse, apps and windows, notifications, clipboard, waits and sessions - Headless by default — ref actions don't move the pointer or take focus — with a global `--headed` flag for physical input - Version-matched agent skill docs built into the binary (`agent-desktop skills get desktop`) - An in-process C library (`libagent_desktop_ffi`) for calling it from other languages, shown with a Python `ctypes` example - Native binaries for x64 and ARM64, installed through npm, Bun, Yarn or pnpm ## Getting started agent-desktop supports macOS 13 or later, Windows 10 1809 / Windows Server 2019 or later, and Ubuntu 24.04 or later. On macOS it needs Accessibility permission; on Linux it needs a graphical session with `at-spi2-core`. 1. Install the CLI Install the package globally (Bun, Yarn and pnpm work too), then check the version and what your platform supports. ```bash npm install -g agent-desktop agent-desktop version agent-desktop status ``` 2. Grant permissions on macOS macOS requires Accessibility permission for the terminal that runs agent-desktop. The built-in guide walks you through it. Windows needs no extra grant for apps at the terminal's integrity level. ```bash agent-desktop permissions --guide ``` 3. Observe, act, verify Take a snapshot of an app's interactive elements, act on a ref from that snapshot, then snapshot again to confirm the change. Refs are tied to the snapshot they came from. ```bash agent-desktop snapshot --app Finder -i # interactive elements, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot --app Finder -i # observe again to verify ``` 4. Hand it to your agent Load the version-matched skill guide into your agent first, list the running apps, and map dense apps with a compact skeleton before drilling into a region. ```bash agent-desktop skills get desktop agent-desktop list-apps agent-desktop snapshot --skeleton --app Slack -i --compact ``` ## When to use it - Letting a coding agent such as OpenCode write a note in a native text editor, save it, and confirm the file in Finder - Automating desktop apps that have no API, by acting on their accessibility elements instead of screen coordinates - Giving an agent the same command set on a Mac, a Windows machine and a Linux desktop - Calling desktop automation in-process from a Python or other program through the C library ## Version history - 2026-11-03 v1.1.0: Fixes the Windows overlay (crashes at exit, double scaling, a black flash at startup) and lets macOS reach the Dock, menu bar items and system panels. Adds `type --interval-ms` for steady typing and more reliable ref re-identification on all three platforms.