dsh-computer-use-plugin

Most Computer Use agents lean on Accessibility APIs or UI trees to locate elements, which cripples cross-platform reach and leaves the LLM guessing coordinates on a full-screen image. Clicks land off-target yet the agent reports success; buttons must be re-found from scratch every session. dsh-computer-use-plugin eliminates the UI-tree dependency entirely, adopting a pure Vision-Only Grounding architecture: the AI perceives the interface from screenshots alone and drives the desktop via coordinates, working natively on Windows, macOS, and Linux, as well as RDP sessions, cloud sandboxes, and even game windows.

Four self-built engines target the four real failure modes of vision-only CUA agents. Effect verification captures full-screen dHash fingerprints before and after each action to flag no-ops. Two-stage grounding (zoom_inspect) crops and magnifies the target neighborhood for pixel-accurate back-mapping. Scene-based UI memory persists verified clicks as landmarks and recalls them across sessions via natural-language queries. The action journal records every step as JSONL, letting a successful sequence be replayed as an executable macro. A semantic-closure layer adds local OCR text detection and visual diffing, pushing verification from the pixel layer to the semantic layer: the system confirms whether expected text actually appeared on screen and pinpoints exactly which region changed. The latest iteration introduces a self-evolving skill library that auto-distills successful trajectories into reusable, reliability-scored skills, and a credential risk gate that mandates human input for sensitive fields.

Built for developers constructing autonomous desktop agents, engineering teams that need cross-platform UI automation, and researchers exploring vision-based CUA. Licensed under MIT, the plugin follows the DSH plugin spec for zero-config installation, and also supports source-level hot-loading and a dry-run mode for risk-free experimentation.