computer-control

When an AI agent needs to actually drive a desktop computer, it requires a reliable and safe low-level channel. computer-control fills that gap for dsh agents: it lets the model capture the screen for observation, inject precise mouse and keyboard events, and perform semantic actions through the Windows UIA accessibility tree rather than relying solely on brittle pixel-coordinate clicks. The plugin provides a complete capability set, including full-screen or region screenshots with scaling and grayscale, pointer move/click/double-click/triple-click/drag/scroll, single-key and chord injection with arbitrary Unicode text input, hierarchical accessibility-tree snapshots and semantic activation, plus batch execution to reduce model round trips. The coordinate system uses a canvas-scaling mechanism that correctly handles multi-monitor and per-monitor DPI scenarios.

Safety is a core design principle. A triple-trigger emergency stop (global hotkey, protocol command, and panic file) guarantees operations can be halted at any moment; an allow/deny rule engine provides fine-grained control by tool name and arguments, with deny rules always taking precedence; high-risk actions trigger a human-confirmation flow with automatic denial on timeout; an idle-standby mechanism automatically pauses the session after a configurable inactivity threshold; and a dry-run mode logs every action without touching real hardware, making it ideal for safe rehearsals and integration debugging.

The plugin is aimed at developers and teams building desktop-automation workflows on Windows 10/11 within the dsh framework, as well as anyone who wants to rehearse agent behavior in a zero-risk environment, validate operation logic, and confirm system integration before going live. It integrates with dsh via standard JSON-RPC over stdio, is self-contained and plug-and-play, and requires no modification to the agent core.