dsh-computer-use

The dsh-computer-use plugin addresses the fundamental limitation of pure-text large language models like DeepSeek Harness, which lack the ability to directly interact with a computer's desktop. These models can process text but cannot see the screen or control input devices. This plugin bridges that gap with an elegant closed-loop system: it captures screen visuals via screenshot or screen recording, analyzes them using the Xiaomi MIMO vision model for intelligent understanding, and then executes real mouse, keyboard, window, and program controls. Beyond simple step-by-step commands, it can autonomously loop through observe-analyze-act cycles, dynamically adjusting actions based on real-time observations until a task is completed.

Its core capability lies in the deep integration of multimodal vision with desktop automation, offering a complete solution from perception to action. This plugin is ideal for AI developers and automation enthusiasts who wish to explore or implement language-model-driven desktop automation (RPA). Whether you need AI to handle repetitive computer tasks, test user interfaces, or simply are interested in AI controlling a real computer, dsh-computer-use provides powerful support. It simplifies the complex pipeline from visual understanding to physical manipulation, enabling non-graphical models to operate in real-world environments and opening new avenues for developing intelligent agents and automated workflows.