Introduction¶

In DeepSeek Harness (DSH) for desktop automation, common approaches involve calling system APIs or relying on screenshots plus coordinate clicks. The former is not model-friendly, while the latter can easily lead to misaligned clicks and struggles to express the “observe first, then act” workflow. dsh-computer-use is a community-maintained workflow plugin that complements harness-desktop with “virtual cursor human-like operations”: the model first observes the screen, then performs sliding, clicking, and typing through an independent cursor, with visible processes, supporting macOS / Windows / Linux.

This article introduces the plugin’s positioning, 11 tools, installation steps, and usage considerations.

What It Is¶

• Name: dsh-computer-use (GitHub: 988hj7tczd-oss/dsh-computer-use)
• Maintainer: 988hj7tczd-oss
• Category: Workflow (SkillHub directory page labeled “Installable · Workflow”, current version v0.1.0)
• License: MIT
• Engine: Based on cua-driver (MIT, MCP standard interface)

The plugin registers 11 model-friendly tools for DSH. The core design involves using the accessibility (AX) tree to “see” the screen and locate element coordinates (zero visual token cost); all operations go through a pixel-level virtual cursor with sliding animations and realistic clicks, simulating human operations.

Core Features¶

Observation and Operation Tools¶

Security Design¶

1. Virtual Cursor Isolation: Operations go through cua-driver’s independent Agent cursor, not preempting the real mouse.
2. Observation Snapshot TTL: Snapshots expire in about 15 seconds; actions are rejected after expiration, requiring a re-screen_observe.
3. Region Restrictions: Optional allowedApps whitelist; operations on apps outside the list are rejected.
4. Dangerous Operation Approval: When target labels contain danger words like “delete / payment / transfer / logout”, the DSH approval service is consulted to ask the user.
5. Sensitive Input Protection: Password fields (AXSecureTextField) reject automatic input.
6. No Snapshot Rejection: Any action must be preceded by screen_observe, eliminating blind operations.

Coordinate mode (x/y) and untargeted input (computer_type / computer_key landing in the foreground app) cannot predict target content, so security fallback relies on snapshot TTL and operation visibility; computer_key does not validate the shortcut itself, so do not grant access to untrusted models.

Optional Capabilities¶

Visual Fallback: For interfaces without an AX tree, such as games or Canvas, set ZHIPU_API_KEY to allow screen_observe to degrade to visual understanding (e.g., glm-4.6v-flash, see README) when AX is empty.

Cursor Theme: Built-in rainbow gradient pointer theme com.dsh.computeruse.rainbow; configurable via cursorTheme, falling back to the engine default cursor if not installed.

Installation and Enabling¶

Prerequisites¶

• harness-desktop (including dsh rc runtime)
• cua-driver installed and authorized (macOS: Accessibility + Screen Recording; Windows: normal user permissions)
• The plugin defaults to looking up cua-driver in PATH; if not in PATH, set CUA_DRIVER_BIN=/path/to/cua-driver

One-Click Installation¶

The repository provides install.sh, which injects into the home-level user patch layer without modifying profile configuration:

# Dry run
./install.sh --dry-run

# Install
./install.sh

# Uninstall
./uninstall.sh

Restart harness-desktop after installation for it to take effect.

Windows / Linux users: install.sh defaults DSH_HOME to the macOS path, so first run export DSH_HOME=<your dsh home directory> before executing; or manually in two steps:

1. ln -sfn <plugin directory> "$DSH_HOME/profiles/web/node_modules/dsh-computer-use"
2. Insert plugin registration in $DSH_HOME/cordis.patch.yml (see repository cordis.patch.yml)

Optional Configuration¶

Override in $DSH_HOME/cordis.patch.yml:

- id: dsh-computer-use
  config:
    ttlMs: 15000        # Snapshot validity period (milliseconds)
    maxElements: 500    # Maximum numbered elements for screen_observe
    allowedApps: []     # Region restriction whitelist (empty = no restriction)
    cursorTheme: com.dsh.computeruse.rainbow  # Virtual cursor theme (empty = engine default)

Visual mode (optional):

export ZHIPU_API_KEY=your_key

Keys can be applied for at the Zhipu Open Platform: https://open.bigmodel.cn

Typical Usage¶

For development verification, an isolated profile can be used without modifying real GUI configurations. Example from README:

DSH_HOME=$PWD/.dsh-p0 ELECTRON_RUN_AS_NODE=1 \
  /Applications/harness-desktop.app/Contents/MacOS/harness-desktop --expose-internals \
  /Applications/harness-desktop.app/Contents/Resources/app/node_modules/@deepseek-ai/dsh/lib/bin.js \
  --profile test "Please call screen_observe to observe the current window and report"

Daily workflow: First, use screen_observe to get numbered elements and coordinates, then call tools like computer_click; any action must be preceded by a non-expired observation snapshot.

Applicable Scenarios and Considerations¶

Suitable for: Workflows in DSH dialogues where models need to “view the desktop, click the interface, enter text”; scenarios where visible operations and isolation from the real mouse are desired.

Limitations (as stated in README):

• Windows / Linux pending real-device testing (engine officially supported)
• Pixel coordinate calibration is approximate; first-click offset can be calibrated using visual mode or engine screenshot comparison
• Visual screen reading has average precision for small fonts; windows like macOS calculator may not be in the AX tree

Permissions and Security: The plugin runs with the current dsh process permissions. Before installation, read the source code and MIT license to confirm if allowedApps and approval strategies match your environment. SkillHub is a community directory and has no official affiliation with DeepSeek / High-Flyer.

Links¶

• SkillHub Directory Page: https://www.skillhub.cn/plugins/988hj7tczd-oss/dsh-computer-use
• GitHub: https://github.com/988hj7tczd-oss/dsh-computer-use
• npm: https://www.npmjs.com/package/dsh-computer-use