dsh-computer-use-windows

In real desktop automation, coordinate-only clicking is surprisingly brittle: screenshots capture unrelated windows, OCR positions drift, and a click can silently land on the wrong UI state. dsh-computer-use-windows was built to address exactly this failure mode, treating every action as an observable state transition rather than a blind pixel command.

The core design rests on three invariants: window-scoped coordinates, text before pixels, and post-action verification. Capabilities include target-window screenshots with OCR text grounding, an OCR-locate-click-verify-retry loop, mouse-keyboard-clipboard primitives, window enumeration and focus, batch action execution, and an optional OpenAI-compatible vision endpoint. In OCR-only mode screenshots never leave the machine, so no external vision model is required.

The plugin is currently an experimental alpha. It is best suited for teams or individuals developing and testing Windows desktop-automation workflows in controlled environments. While it does not yet promise production-grade unattended automation, it provides a clear engineering skeleton for the see-act-verify pattern that makes desktop automation reliable.