When developing DeepSeek Harness-based agents, allowing an agent to perform operations in a Windows desktop environment is a common requirement. Existing solutions often require installing complex native drivers or SDKs. The datuloar/dsh-computer-use-win plugin provides Harness with screenshot, click, input, and scrolling capabilities through the CLI tool dsh-cu, and includes an on-screen overlay for human observation and intervention.

Plugin Positioning

This is a Windows computer use plugin designed for DeepSeek Harness. It does not depend on external drivers or SDKs, and directly uses the built-in .NET Framework and UI Automation from Windows to perform operations. The CLI tool dsh-cu can run independently or be used together with Harness’s skill system.

Core Capabilities

  • Screen Reading: Converts screen content to text (OCR) or reads the UI control tree, and supports clicking controls by name.
  • Input and Actions: Supports screenshots, mouse movement, click, drag, scroll, keyboard input, and key combinations.
  • Human Supervision: Includes a built-in screen indicator, allowing humans to observe the agent’s operation status in real time and press the ESC key to stop the agent.
  • General-purpose CLI: The command-line tool can be used with any Harness skill or standalone script.

Installation and Enablement

Add the plugin using the official installation command:

dsh plugin --profile web add github:datuloar/dsh-computer-use-win

After installation, the system registers a skill named dsh-computer-use and provides the dsh-cu command-line tool.

Typical Usage

The plugin provides a series of commands for desktop operations:

  • Start the overlay for human observation:
    dsh-cu overlay
  • Screenshot:
    dsh-cu shot
  • Click specific coordinates:
    dsh-cu click 640 380
  • Input text:
    dsh-cu type "hello"
  • Input key combination:
    dsh-cu keys ctrl+l
  • Drag:
    dsh-cu drag 420 300 980 300
  • Click by control name:
    dsh-cu tap "Save"
  • Read screen content:
    dsh-cu read
  • Get control tree:
    dsh-cu tree

Dependencies and Environment Requirements

Running this plugin requires the following conditions:

  • Operating System: Windows desktop environment.
  • Runtime Environment:
    • Node.js 18+.
    • Windows PowerShell 5.1.
  • Notes: PowerShell 7 (pwsh) is not supported by this plugin because the backend relies on the Windows built-in .NET Framework and WinRT rather than the .NET Core runtime.

Limitations and Notes

  • No Browser DOM: The plugin is based on UI Automation, so the information it obtains consists of controls exposed by applications rather than the browser DOM tree.
  • No Multi-Monitor Support: Multi-monitor environments are not currently supported.
  • No Application Whitelist: Operations are not restricted to specific applications.
  • Safety Mechanisms:
    • Input operations are rejected until the overlay has started.
    • The agent cannot close the overlay by injecting the ESC key; it must be stopped manually by a human.
  • Sandboxed Execution: The plugin can run in Harness’s sandbox environment, but it relies on file descriptor capture rather than piped stdio (adapted for Harness’s file sandbox limitations).

Summary

The datuloar/dsh-computer-use-win plugin provides a lightweight, driver-free Windows desktop automation solution. Through the dsh-cu CLI, developers can easily enable agents to interact with local applications, while the overlay mechanism ensures observability and safety.