dsh-desktop

When an AI model can only read page content but cannot click the address bar, switch windows, or drag files, true desktop automation simply does not exist. dsh-desktop solves this by exposing the full desktop capabilities of the host operating system to the model, making the browser chrome, search bar, and any application interactable, and complementing browser extensions that only reach inside the page.

The core toolkit revolves around atomic primitives: a screenshot pipeline that invokes Windows OCR to return per-word bounding boxes so the model can target clicks precisely; left, right, and middle button single and double clicks; press-and-drag gestures; Unicode text input including CJK characters; and arbitrary key combinations such as ctrl+l, alt+tab, and f5. A typical open-url flow is: send ctrl+l to focus the address bar, type the URL, then press Enter. For unknown interfaces the model runs desktop_ocr first to locate coordinates, then clicks.

It suits two audiences: developers who want to grant an AI assistant full desktop control without assembling a heavyweight RPA stack, and automation scenarios that require cross-application operations like dragging files from the explorer into an editor or hopping between windows. The tools run in the host process outside the sandbox, so privileges are high; the model only invokes them when you explicitly issue an instruction.