dsh-computer-use

dsh-computer-use addresses a common gap in agent workflows: a model may understand a task, but it still needs a way to actually operate a desktop interface. By bringing Computer Use capabilities into harness-desktop, it lets AI go beyond text planning and observe the screen, identify interface elements, and drive an independent virtual cursor to move, click, type, scroll, and drag, performing actions much like a human user would.

The plugin’s key design is “real-human operation”: accessibility trees are mainly used for observation and element location, while actions are executed through pixel-level virtual cursor movement. This makes the operation visible and more usable for interfaces that require visual context. It supports screenshot-based observation, zoomed region inspection, multiple viewing modes, and basic input and application control, while adding safety mechanisms such as isolated virtual cursors, snapshot expiration, sensitive input protection, dangerous action approval, and optional app allowlists.

It is best suited for developers exploring or testing desktop agent capabilities, and for advanced users who want to connect visual models, workflows, and real GUI tasks. For scenarios where AI needs to move from describing actions to actually performing them on a computer, this plugin provides a practical entry point.