Introduction

When building DSH agents, the model can usually only act through web page content (Edge extensions) and cannot directly control the browser frame (such as the address bar or search bar) or other arbitrary windows. The dsh-desktop plugin fills this gap by providing OS-level control capabilities, enabling the model to operate the entire browser and any program on the desktop.

What Is This?

This is a Host Plugin maintained by Hyna-hla. Its core value is to provide OS-level desktop control capabilities, allowing the model to control the browser frame, perform window operations, and manage arbitrary applications.

Core Features

The plugin provides the following tools:

  • desktop_ocr: takes a screenshot and calls Windows OCR, returning bounding boxes for each word, used to locate UI elements.
  • desktop_screen: obtains the virtual screen dimensions and origin.
  • desktop_click(x, y, button?, double?): moves the mouse and clicks, supporting left, right, and middle buttons as well as double-clicks.
  • desktop_drag(x1,y1,x2,y2): keeps the mouse button held down and drags from one coordinate point to another.
  • desktop_type(text): types Unicode text, supporting Chinese input.
  • desktop_press(combo): simulates keyboard shortcuts, such as ctrl+l (focus the address bar), enter, alt+tab, f5, ctrl+t, etc.
  • desktop_cursor: gets the current cursor position.
  • desktop_screenshot(path?): takes a screenshot and saves it as a PNG file.

Installation and Enabling

After cloning the repository, run the following commands in order:

pnpm install
pnpm build
.\scripts\install-to-profile.ps1 -Name dsh-desktop -SourceDir .
# 重启 DeepSeek Harness

Typical Usage

Open a URL:
1. Call desktop_press "ctrl+l" to focus the address bar.
2. Call desktop_type "https://..." to type the target address.
3. Call desktop_press "enter" to navigate.

Click Any UI Element:
1. Call desktop_ocr to get the bounding boxes of text on the screen.
2. Based on the returned coordinates, call desktop_click to perform the click.

Principle and Boundaries

  • Runtime mechanism: The plugin spawns powershell.exe in the host process and executes lib/desktop.ps1, using Win32 APIs (such as SetCursorPos, mouse_event, and SendInput) and the OCR included with Windows. It is not subject to DSH sandbox restrictions.
  • Division of responsibilities: It complements Edge extensions: the extension manages web page content, while the desktop layer manages the browser frame and all programs.
  • Permissions and boundaries: The model calls these high-privilege tools only when instructed by you. The model can “see” everything on the screen (OCR) and can also operate everything on the screen.

Cautions

  • The plugin runs with the privileges of the host process. Check the source code and license (MIT) before installing it.
  • This plugin does not conflict with existing Edge extension capabilities; it is a desktop-level complement.

Ecosystem and References

The DSH philosophy is “everything is a plugin.” More plugins are available in the community directory. The plugin source code is on GitHub.