Introduction

In DeepSeek Harness (DSH), enabling an Agent to operate native applications on the local machine typically involves global mouse/keyboard injection or script bridging. This approach often leads to two issues: first, actions become disconnected from the current UI state, causing the model to replay clicks based on outdated interface information; second, input events enter the global HID stream, potentially moving the system cursor, taking focus from the foreground, and interrupting ongoing user operations.

dsh-computer-use is a DSH plugin maintained by Anionex, positioned as a native action layer for macOS. It prioritizes Accessibility observation, binds each action to a non-expired observation, and routes fallback inputs via process/window-directed delivery to the target application instead of the global desktop. It currently supports macOS 14 and later versions.

What Is This

@anionex/dsh-computer-use is a DSH Bundle published on npm, with its GitHub repository at Anionex/dsh-computer-use. The plugin is categorized as admin-security, currently at version v0.1.0, and uses the MIT license.

It solves the core problem of enabling an Agent within a DSH Session to observe the Accessibility tree of macOS applications, perform semantic or fallback inputs on specified processes/windows, and return fresh state after each action. At the same time, it reduces the risk of erroneous operations and user disruption through scoped permissions, expired state rejection, and foreground-safe input strategies.

Core Features

Below are the capabilities explicitly listed in the plugin’s README.

Observe Before Acting

computer_observe returns a bounded Accessibility tree, indexed elements, accurate app/process/window metadata, permission states, and optional screenshot artifacts. Each element carries an observation-local index and an opaque targetHandle; subsequent actions must reference a non-expired observationId.

Foreground-Safe Input Routing

The default strategy intentionally avoids interfering with the user’s current work:

  • It does not move the system cursor; there is no cursor warp path in the helper.
  • Fallback paths for click, scroll, and drag use pid/window-directed SkyLight routing, bypassing the global HID event stream.
  • Semantic Accessibility, keyboardPolicy: preserve, and target process pointer inputs do not activate applications; the Bundle’s default keyboardPolicy: activate only brings the target application to the foreground before keyboard fallback.
  • Click, scroll, and drag use an independent Agent software cursor (click-through, no application activation), leaving the system’s real cursor unchanged.

Prioritize Semantic Input, Fallback to Target

Actions prefer using AXPress, editable values, selected-text assignments, and Accessibility actions declared by elements; pointer or keyboard fallback is only used when necessary. Keyboard input is sent to the selected pid; pointer input carries window-local coordinates and is sent to the selected pid and CGWindowID.

Scoped Permissions and Sensitive Action Confirmation

Read/control leases are separated by Agent, Session, turn, and exact bundle id. When no grant is configured, DSH requests approval: read approval is valid within a Session, while control approval is only valid for the current turn. High-impact actions require one-time confirmation.

Model Tools Exposed on Demand

The Bundle initially only contributes computer_use_activate. After the current Agent loads the Computer Use Skill, it exposes execution tools such as computer_list_apps, computer_observe, computer_click, computer_set_value, computer_type_text, computer_press_key, computer_scroll, computer_drag, computer_perform_action, computer_wait, and computer_confirm.

Installation and Enabling

Prerequisites

  1. macOS 14 or later.
  2. DeepSeek Harness with Web or Headless Profile installed and Skill Tool mounted.
  3. macOS Accessibility permission (required for observation and native actions).
  4. macOS Screen Recording permission is only needed when requesting screenshots.

The published package name is @anionex/dsh-computer-use. The old name @dsh-external/dsh-computer-use was never published to npm and cannot be installed; please update old profile or manifest references before installation.

Installation Commands

Below are the official installation methods from the README, to be added to web and headless profiles respectively:

dsh plugin --profile web add @anionex/dsh-computer-use
dsh plugin --profile headless add @anionex/dsh-computer-use

After installation, you can confirm the configuration is written with the following commands:

dsh --profile web --dump-config | grep computer-use
dsh --profile headless --dump-config | grep computer-use

After modifying an installed plugin, you need to restart the running dsh web host and then create a new Session for the host to reload the Bundle and Skill catalog.

Loading the Skill

In a new Session, enter:

/computer-use

Typical Usage

After loading the Skill, you can have the Agent operate native applications following the “observe first, then act, return fresh state” flow. The README provides the following example:

Use Computer Use to check a running DSH Computer Use Fixture, enable the deterministic option, and report results based on the fresh state returned after actions. Prioritize Accessibility elements and avoid reusing old observations.

The repository’s built-in AppKit fixture is used to verify foreground-safe strategies: the published test starts the fixture in background mode via open -g, then completes the observe → click → fresh observation flow using the same protocol as the Agent, and checks that activation is not-requested and pointerRouting is target-process.

A typical workflow can be summarized as:

  1. computer_list_apps lists bounded user applications and permission diagnostics.
  2. computer_observe obtains the target application’s Accessibility tree and observationId.
  3. Based on element index or targetHandle, call tools like computer_click or computer_set_value.
  4. After the action settles, it returns a full or differential observation for the next decision.

Use Cases and Considerations

Who Is It For

  • Developers who need to operate macOS native applications within a DSH Agent when no dedicated API/CLI plugin is available.
  • Scenarios where the Agent operates other applications in the background while minimizing interference with the user’s current foreground work.
  • Admin-security workflows that require stale-state rejection, per-application leases, and sensitive action confirmation.

What It Does Not Replace

The README clearly defines its scope: dsh-computer-use is a native action layer and does not replace more narrow interfaces:

  • Browser tasks should continue to use browser automation and DOM/CDP state.
  • When APIs, CLIs, or dedicated application plugins are available, they should be used instead.
  • OCR and visual understanding should be handled by separately installed dsh-vision-toolkit, processing screenshot artifacts via the vision-tools Skill.

Before Installation

The plugin runs with the current DSH process permissions, requires Accessibility permission, and additionally requires Screen Recording permission for screenshots. Before installation, you should read the GitHub repository source code and MIT license to ensure the interaction strategies and permission models meet your security requirements. The helper executable is a DSH internal transport implementation and should be used through registered tools to preserve application leases, sensitive action confirmations, and host policy checks.

Currently, only macOS is supported; this plugin cannot be used in Linux or Windows environments.

Conclusion

dsh-computer-use combines macOS Accessibility observation, expired state rejection, scoped permissions, and foreground-safe input routing into DSH’s native action layer, making it suitable for scenarios requiring reliable and controllable Agent operations on the local GUI.