agent-desktop

// Overview

agent-desktop is an open-source tool from Vercel Labs that gives an AI agent hands on a desktop computer. It is a command-line program and a C library that read the user interface of native apps on macOS, Windows and Linux and act on it: click, type, scroll, manage windows, read notifications and the clipboard. It is a tool that agents call, not an agent itself, and it contains no model. For web pages, the same team points you to its sibling project, agent-browser.

The key design choice is the accessibility tree. Instead of matching pixels in screenshots, agent-desktop reads each app through the operating system's own accessibility APIs — the AX API on macOS, UI Automation on Windows and AT-SPI on Linux. A snapshot returns JSON with a snapshot ID and a ref for every element an agent can act on, and actions target those refs (for example `@s8f3k2p9:e8`). The trade-off is stated plainly: an app that publishes no accessibility tree is invisible to it.

Every CLI call runs as its own process, and the reasoning loop stays in the agent: observe with a snapshot, act on a ref, then snapshot again to verify. Errors such as `STALE_REF` and `AMBIGUOUS_TARGET` tell the agent to look again. Ref actions are headless by default, so they do not move the pointer or steal focus; a global `--headed` flag switches to physical input. The binary also ships its own skill docs matched to its version, which an agent can load with `agent-desktop skills get desktop`.

// What it does

• Reads native desktop apps through the OS accessibility tree on macOS (AX API), Windows (UI Automation) and Linux (AT-SPI) — no pixel matching
• JSON snapshots with a snapshot ID and a ref per actionable element; bare refs like `@e8` are rejected so stale targets are caught
• About 60 commands with the same names on every platform: observation, ref actions, keyboard and mouse, apps and windows, notifications, clipboard, waits and sessions
• Headless by default — ref actions don't move the pointer or take focus — with a global `--headed` flag for physical input
• Version-matched agent skill docs built into the binary (`agent-desktop skills get desktop`)
• An in-process C library (`libagent_desktop_ffi`) for calling it from other languages, shown with a Python `ctypes` example
• Native binaries for x64 and ARM64, installed through npm, Bun, Yarn or pnpm

// Getting started

agent-desktop supports macOS 13 or later, Windows 10 1809 / Windows Server 2019 or later, and Ubuntu 24.04 or later. On macOS it needs Accessibility permission; on Linux it needs a graphical session with `at-spi2-core`.

npm install -g agent-desktop
agent-desktop version
agent-desktop status

agent-desktop permissions --guide

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify

agent-desktop skills get desktop
agent-desktop list-apps
agent-desktop snapshot --skeleton --app Slack -i --compact

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Letting a coding agent such as OpenCode write a note in a native text editor, save it, and confirm the file in Finder
• Automating desktop apps that have no API, by acting on their accessibility elements instead of screen coordinates
• Giving an agent the same command set on a Mac, a Windows machine and a Linux desktop
• Calling desktop automation in-process from a Python or other program through the C library

// How agent-desktop compares

agent-desktop alongside other open-source computer & browser use tools AI/TLDR tracks, ranked by GitHub stars.