dsh-desktop-automation

Browser automation frameworks like Playwright can drive web pages with ease, yet the desktop apps where most real work happens—video editors, design tools, office suites, system settings—remain entirely out of reach for AI agents. dsh-desktop-automation closes that gap: it gives dsh agents the ability to see the screen, move the cursor, type on the keyboard, and operate any native macOS application just like a human would.

The plugin registers 14 agent-callable tools spanning mouse movement, clicking, dragging, scrolling, key presses, text input, app and window management, plus full-screen and region screenshots. The standout feature is the visual closed loop: a screenshot is sent to a vision model for recognition, a grounding pass returns precise coordinates of interactive elements, the agent clicks the target, and a follow-up screenshot confirms the result—completing a full recognize, locate, click, verify cycle. This works even on apps whose accessibility tree is unavailable, since positioning relies on vision rather than the a11y tree.

It is ideal for teams automating batch renames, cross-app data transfers, and repetitive form-filling; for QA engineers running GUI test-and-verify cycles on desktop software to replace manual spot-checks; and for developers who want to remotely direct a Mac to perform desktop tasks. As long as dsh already has vision credentials configured, the plugin works out of the box with no extra API keys required.