dsh-mac-cua

In the DSH agent ecosystem, the command line is well-served, but the desktop GUI has always been the missing piece. dsh-mac-cua closes that gap: it lets the agent read and operate any macOS desktop application through the native Accessibility API, from Finder to Safari to any Cocoa or Swift app, without AppleScript, JXA, or System Events. Everything flows through a single, fixed JavaScript REPL channel.

The design philosophy is deliberately minimal: only two tools are exposed, js and js_reset. Every capability, including clicking, typing, pasting, key chords, scrolling, dragging, selecting text, reading the accessibility tree, and capturing screenshots, lives inside the sky API. The first read returns the full tree; subsequent reads return only the diff, keeping long interactions cheap in tokens. Elements are addressed by element_index with coordinates as a fallback rather than a default. Paste restores your clipboard automatically, and multi-line text should use paste rather than type_text to avoid unintended form submission.

This plugin is ideal for developers who run DSH Desktop and want their AI agent to operate macOS GUIs the way a human would. It requires macOS 14.4+ and Node.js 20+, with Accessibility permission granted and Screen Recording permission if screenshots are needed. Once set up, the agent gains the same class of desktop read-write access as a screen reader, with prompt-level policy guidance to confirm before high-risk actions such as deleting data, sending messages, or submitting financial transactions.