Foreword

DeepSeek Harness (DSH) uses a plugin-based architecture, allowing developers to extend core functionality. dsh-computer-use is a standalone Cordis bundle maintained by leolee9086. It provides DSH with visual, native accessibility, and Windows Narrator operation paths. Version 0.5.0 registers 18 computer_* tools. Accessibility observation, reading, and semantic actions can be used independently, and text models can also operate native applications.

Core Features

This plugin mainly covers the following capabilities:
* 18 computer_* tools: Covering visual, pointer, keyboard, accessibility, window, and Narrator operations.
* Independent accessibility loop: Does not depend on image capabilities; performs element discovery, reading, and control solely through accessibility interfaces.
* Windows native runtime validation: As the primary backend, supports UIA and MSAA.
* Cross-platform contract tests: macOS AX and Linux AT-SPI are implemented and contract-tested.
* Background capture: Supports PrintWindow to capture target windows without activating the windows.
* Capability coverage: Includes Pointer, Keyboard, Accessibility, Window, and Narrator.

Installation and Enabling

In environments where Harness CLI is available, add the repository or packaged artifact as a profile bundle. The installation command is as follows:

pnpm dsh plugin --profile web add <path-to-dsh-computer-use>

The bundle patch simultaneously activates the host computer service, tools, and workflow prompts, without requiring edits to agent presets. After installation, restart the Harness entry point to load the new host code.

Typical Usage

Independent Accessibility Loop

This path does not require image capabilities. The operation sequence is as follows:

  1. Call computer_windows with operation:"list" to obtain a short-lived window_id; you can also directly observe the current focused application.
  2. Call computer_accessibility. On Windows, you can specify window_id; backend:"native" defaults to UIA, and backend:"msaa" is the legacy implementation.
  3. Use computer_find to narrow the scope of elements, or directly use the element_id from the snapshot.
  4. Use computer_read to read bounded content.
  5. Use computer_element to operate via the patterns exposed by the control.
  6. Re-observe after each control attempt before performing the next action.

Visual and Input

  • Visual: Use computer_screenshot, computer_find_image, and computer_click_image. Windows supports multiple monitors, region/zoom, foreground window capture, explicit background PrintWindow, template matching, and unique-match clicking. Images are returned as DSH attachments.
  • Pointer and keyboard: Use computer_move, computer_click, computer_drag, computer_scroll, computer_key, computer_type, and computer_input. Windows supports triple-click, modifiers, hold, path-based dragging, extended keys, and input sequences.

Windows Narrator

Calling computer_narrator(operation:"status") only checks whether the Narrator process exists in the current session and does not launch Narrator. Commands such as item/view navigation, reading the current item/window/title/document/selection/line/word, and toggling scan mode are supported. Responses are explicitly marked as unknown / Standard assumption.

Platform Capabilities

Capability Windows macOS Linux
Desktop screenshot/basic input/window listing and focusing Verified in practice Implemented and contract-tested Implemented and contract-tested; input/windows require X11 xdotool
Independent accessibility observation/reading/basic actions UIA and MSAA verified AXValue / AXSelectedText; requires Automation + Accessibility AT-SPI text/range/selection/caret/numeric; requires Python + pyatspi + AT-SPI bus
Region/window screenshot/image matching Verified in practice Explicitly unsupported Explicitly unsupported
Atomic input sequences/path dragging/extended hold/repeat Implemented and core real-device check Explicitly rejected Explicitly rejected
PrintWindow/child HWND/window management/Narrator Windows-only Explicitly rejected Explicitly rejected

Dependencies and Notes

  • Environment requirements: Node >= 22.19, pnpm; Rust >= 1.88.
  • Cross-platform snapshots: macOS/Linux snapshots currently come from the focused application and do not support Windows-style target-window semantic roots.
  • Dependencies: Windows accessibility/Narrator depends on optional edge-js; C# is compiled and invoked inside the Node process. Visual/window input uses the repository-provided Rust executable.
  • Approval modes: Default observeApproval / controlApproval is ask. When the authoritative DSH permission preset is Full access, prompt-free behavior is inherited.
  • Cleanup mechanism: Forcibly terminating a process cannot guarantee that destructor cleanup runs.
  • Uncertain facts: There is currently no task success-rate benchmark and no SOTA-equivalent conclusion.

Summary

dsh-computer-use provides a complete set of computer operation tools, with a focus on verifying native capabilities on Windows and completing contract tests on macOS and Linux. It is suitable for development scenarios where text models directly operate desktop applications and have a strong dependency on accessibility interfaces.