Introduction

The DeepSeek Harness (DSH) agent must have the ability to “see” and “act” when handling desktop application tasks. Existing DSH profiles may lack specific interaction mechanisms for the Windows desktop. sage-guikit is a GUI toolkit designed specifically for DSH. It packages Windows desktop control capabilities into a profile bundle, providing the agent with screen layout monitoring, screenshots, mouse/keyboard operations, window management, and UI Automation-based structured querying and validation.

Core Features

The plugin provides 13 model tools, with core capabilities including:

  • Screen and Screenshot: Retrieves the display layout contract, supports full-screen and single-window screenshots, and includes coordinate annotation features.
  • Interaction Operations: Supports clicking, dragging, typing, key presses, and scrolling.
  • Window Management: Lists windows and gets/activates/moves windows.
  • Locating and Validation: Locates controls by element name or UIA tree, supporting polling validation and semantic validation.

Installation and Enablement

Installing the plugin requires manual operation and needs to be introduced as a DSH profile bundle.

  1. Enter the DSH profile directory (for example, ~/.dsh/profiles/web).
  2. Run the installation command:
    pnpm add dsh-guikit
  1. Edit the package.json in that directory and add "dsh-guikit" to the dsh.profile.bundles array.
  2. Restart the DSH Web service.

For local development, you can also use the link: method, but note that bare imports may not resolve to the profile’s node_modules. In that case, install the peer dependencies (@deepseek-ai/cordis, @deepseek-ai/dsh-tools) in the plugin directory.

Tool Overview

The plugin registers the following model tools:

Screen and Screenshot

  • gui_screen: Retrieves display layout, virtual desktop origin, DPI scaling, and other information. Always call this tool before any coordinate operation.
  • gui_screenshot: Captures the full screen or a specified region and returns the image path. When annotate is enabled, it draws rulers, grid coordinate labels, and a watermark in the bottom-left corner.
  • gui_window_shot: Captures a single window (PrintWindow), without activating the window and without obstructing it. Returns coordinates in window-local space.

Interaction Operations

  • gui_click: Clicks at the specified coordinates. Supports the window parameter and the space parameter (window-local). Optionally includes the verify parameter for post-click validation.
  • gui_drag: Simulates holding down the mouse to drag, used for selecting text, dragging sliders, and similar tasks.
  • gui_type: Types into the focused control, supporting unicode (bypass IME) and clipboard modes.
  • gui_key: Sends system keys or key combinations (such as ctrl+s).
  • gui_scroll: Scrolls the wheel at the specified location.

Locating and Managing

  • gui_locate: Locates an element by element name or window title. Does not guess coordinates, and returns clickable coordinates only when status="found". Supports timeoutMs polling.
  • gui_uia: Uses Windows UI Automation structured queries, supporting operations such as tree, find, invoke, and value.
  • gui_window: Window management, supporting list, rect, activate, and move operations.

Validation

  • gui_wait: Polling validation, supporting pixel change comparison or waiting for a window to appear.
  • gui_verify: Semantic validation, asserting whether an element exists or whether an attribute satisfies a condition. Returns a ternary result (satisfied / unsatisfied / unknown).

Coordinate System and Contract

The plugin defines three spaces. Tools do not guess for each other, and conversions must strictly follow the returned contracts:

Space Name Purpose Origin
virtual-desktop-physical Input parameters for gui_click / gui_drag; return values from gui_screen / gui_locate Top-left of the virtual desktop
image-pixels (screenshot) Image pixels read by read_image Top-left of the captured region
window-local Return value from gui_window_shot; gui_click input parameter with space="window-local" Top-left of the window

Conversion Logic:
* Image pixels → desktop coordinates: absolute = origin + image pixels × scale
* Desktop coordinates → image pixels: image = (absolute − origin) ÷ scale

Notes: The image returned by read_image is usually downsampled (for example, ×1.50), and pixel spacing is unreliable. Be sure to read coordinates from the label text in images generated with annotate=true, rather than calculating them manually.

Notes

  • Platform Limitation: Supports Windows 10/11 only.
  • Dependency Environment: Depends on PowerShell 7 and .NET’s System.Drawing / UIAutomationClient.
  • Permissions and Security: Tools run under the permissions of the current DSH process. Check the source code and license before invoking them.
  • Validation Result: gui_verify returning unknown does not indicate success. It means “cannot be determined” and must never be treated as a successful reading.

Conclusion

sage-guikit solves the problems of inaccurate locating and difficult validation in DSH agents on Windows desktop automation by providing structured UI Automation queries and explicit coordinate contracts. After installation, the agent gains complete “eye-hand” capabilities. More details can be found in its GitHub repository: gezi-wen/sage-guikit.