Vercel Labs Open-Sources agent-desktop for Agent Computer Use

Vercel Labs has released agent-desktop, an open-source CLI for AI agent computer use on macOS, Windows and Linux.agent-desktop reads each app through the operating system’s accessibility tree, not through screenshots.An agent gets a JSON snapshot with a ref for every element it can act on.Version 1.

0.2 shipped on October 30, 2026, under Apache-2.0.For developers, any shell-capable coding agent can now drive native apps without a vision model.TL;DR Library: Rust CLI and a C library.60 command names, 56 operational.Runs on: macOS 13+, Windows 10 1809+ or Server 2019+, Ubuntu 24.

04+, on x64 and ARM64.Plain CPU, no GPU.Performance: Deterministic, retry-safe ref actions.No independent computer-use benchmark yet.Best: 78% to 96% fewer tokens on dense Electron apps via skeleton traversal (self-reported).What is agent-desktop?

agent-desktop is a Rust CLI and C library from Vercel Labs for reading and operating desktop apps.It is a tool that agents call, not an agent.It embeds no LLM and has no GUI.Every command prints one JSON envelope and exits.The observe, act and verify loop stays in your agent.

How does agent-desktop work?Each command flows through a fixed chain.The CLI or C library calls agent-desktop-core, which reaches the OS only through a PlatformAdapter trait.

One platform crate then talks to the native accessibility API: macOS: AX API Windows: UI Automation Linux: AT-SPI Core never imports a platform crate.Each binary is compiled to drive exactly one OS.Snapshots and refs A snapshot assigns qualified refs such as @s8f3k2p9:e8.

The snapshot ID inside the ref makes it self-describing.A bare @e8 is rejected with INVALIDARGS.When the UI changes, an action returns STALEREF or AMBIGUOUSTARGET.The agent then takes a fresh snapshot.Every error carries a disposition.Agents should retry only when it says safe.

Ref resolution is strict and fails closed rather than guessing.Skeleton and drill-down Dense apps can expose thousands of accessibility nodes.A --skeleton snapshot reads only 3 levels deep.Cut containers report a childrencount and get a drill ref.

snapshot --root then reads one region into the same snapshot.The builder reported a 78% to 96% token cut versus full-tree dumps in Slack, VS Code and Notion.In Apple Maps, he reported 3,811 tokens per look falling to 273.A simulated 52-step task dropped from 198k to 31k tokens.

These are self-reported figures, not an independent benchmark.Headless by default Ref actions go through the accessibility API without moving the pointer or taking focus.That is what lets agents work while you keep using the machine.The global --headed flag allows physical input when needed.

Every ref action also auto-waits up to 5,000 ms for a visible, enabled, stable target.Trace a commandStep Map, drill, act, verify.Click a step.Example output is trimmed from the official docs (GNOME Calculator).

1Skeleton

2Drill

3Act

4Verify Next step → Builder-reported numbers for skeleton drilling, next to Anthropic's documented screenshot cost.

Apple Maps, tokens per look (self-reported)

Full window snapshot3,811

3,811 Skeleton, then drill273 Simulated 52-step task, total tokens (self-reported)

Full window every step198k

198k Skeleton, then drill31k

31k Reference: pixel-based computer use

One Claude computer-use screenshot (Anthropic docs)~1,000 to 1,800

~1.8k max ↻ Replay

Builder also reports 78% to 96% savings on Slack, VS Code and Notion. No independent benchmark yet. Same 60 command names everywhere. Click a platform for requirements.

macOS

AX API

macOS 13+, Intel or Apple Silicon. Grant Accessibility permission with permissions --guide. Shell surfaces still planned. Windows

UI Automation

Windows 10 1809+ or Server 2019+, x64 or ARM64. No grant for apps at your terminal's integrity level. Higher-integrity apps refuse input. Linux

AT-SPI

Ubuntu 24.04+, glibc 2.35 floor, x86_64 or aarch64. Needs at-spi2-core. Mouse input and cursor overlay need X11. one install for all 3 npm install -g agent-desktop agent-desktop skills get desktop

Where it falls short. Click to expand.

<div class="lim&quo