Preface¶

The core philosophy of DeepSeek Harness (hereinafter referred to as DSH) is “Everything is a plugin”: models, tools, Skills, sessions, sandboxes, storage, and interfaces can all be replaced or reorganized. Agents are already quite comfortable modifying code and running commands in the terminal, but once tasks involve native desktop applications—clicking buttons, filling out forms, confirming status in background windows—the common practice becomes taking a screenshot, guessing coordinates, and then injecting mouse and keyboard events into the entire global desktop. As soon as the interface changes, the old screenshot becomes invalid; if the cursor is moved away or the foreground application is taken over, they will interfere with each other even when a user is working on the same machine.

dsh-computer-use takes a different approach: it first reads the macOS accessibility tree, then binds actions to unexpired observation results, and tries to deliver clicks and inputs to the selected process rather than the entire desktop. This article is organized after cross-checking the community plugin directory, GitHub repository README, and package.json. It should be noted that the community directory deepseek-harness-plugin.com is an independent site and has no official affiliation with DeepSeek / HyperG AI. It should not be treated as an official app store.

What is This¶

dsh-computer-use is a tool and capability plugin for DeepSeek Harness, maintained by Anionex, with the npm package name @anionex/dsh-computer-use and MIT license. The main language of the repository is TypeScript, with a macOS native helper implemented in Swift. As of 2026-08-17, the GitHub repository has approximately 21 stars; when listed on the community directory page, it showed 19 stars. The version in package.json is 0.1.0, and the README clearly states that this is an early version, and both the observable behavior of the model and the provider behavior may change before a stable release.

The problem it solves can be summed up in one sentence: providing native macOS computer control for DSH agents, by default not moving the system cursor, not stealing the foreground focus with pointer actions, and pinning each operation to “fresh, unexpired” Accessibility observations.

The current provider only supports macOS 14 and above (universal binary for both arm64 and x86_64). Windows UI Automation and Linux providers have not yet been implemented. It positions itself as a native action layer and does not intend to replace browser automation, application API/CLI, or standalone visual toolkits.

Core Features¶

Observe First, Then Act¶

The plugin first selects the correct bundle ID and PID, obtains read permissions divided by application, and then returns a bounded Accessibility tree: indexed elements, process/window metadata, permission status, and optional screenshot Artifacts. Each element carries both a compatible index within this observation and an opaque targetHandle. Actions are not aimed blindly at the entire screen, but at specific targets in this observation.

Observations are discrete snapshots captured per request, not a real-time desktop stream. A successful action will go through a bounded settle period before returning a new full or differential observation, allowing the model to verify “what the interface looks like after clicking”.

Expired States Are Directly Rejected¶

Each observation carries an opaque ID and expiration time. Actions must be bound to accurate, unexpired observations; reusing expired observations will be rejected instead of continuing to click using the old tree. The configuration item observationTtlMs controls how long this observation can be reused: the default value 0 means expiration is disabled, and it can also be set up to a maximum of 24 hours. Once Settings pass verification and replace the current provider generation, existing observations and pending confirmations will be invalidated together.

Element positioning is also conservative. When only passing an index, an accurate locator is maintained; low-risk actions can carry a targetHandle and set allowRebind: true. Before inputting text, fresh Accessibility status will be retrieved, and the original locator, unique native identifier (such as AXIdentifier), and unique semantic matching based on role, accessible name, declared actions, and ancestor fingerprint will be checked in sequence. If there is a mismatch or insufficient confidence, COMPUTER_TARGET_AMBIGUOUS or COMPUTER_TARGET_LOW_CONFIDENCE will be returned instead of guessing a button.

Semantic Input First, Pointer Only as Fallback¶

Clicking prioritizes AXPress; editable controls use computer_set_value to directly modify the Accessibility value without going through the clipboard; text insertion prioritizes Accessibility, with the keyboard only serving as a process-targeted fallback. Accessibility actions declared by the element itself can be executed via computer_perform_action.

Mouse, scroll, and drag operations will only be used when necessary, and are delivered to the selected PID and CGWindowID, using window-local coordinates instead of global HID event streams. The README states that there is no system cursor warp path in the helper; target process pointer delivery relies on dynamically resolved SkyLight SPI, and if this route is unavailable, it will fail directly instead of falling back to global injection.

Do Not Steal Cursor or Foreground by Default¶

The default interaction policy is:

interaction:
  focusPolicy: preserve
  keyboardPolicy: activate
  pointerInputPolicy: targeted
  cursorVisualization: visible
  cursorMotionMs: 180
  cursorAutoHideMs: 0

The meanings can be read against the repository documentation:
- focusPolicy: preserve: Pointer actions do not bring the target application to the foreground by default.
- keyboardPolicy: activate: The target application will be activated first before keyboard fallback and press-key to ensure input reaches the correct window; this is the Bundle default and a compatibility choice that may interrupt foreground work. If set to preserve, keyboard events will not activate the window either.
- Clicks, scrolls, and drags will display an independent Agent software cursor (click-through, does not activate applications), while the system’s real cursor remains stationary. cursorVisualization can be set to hidden when visual feedback is not needed.
- pointerInputPolicy: deny will turn off coordinate clicks/fallback, scrolling, and dragging.

The repository includes a deterministic AppKit fixture and a standalone native monitor: the test open -g starts the fixture in the background, and the default path requires that activationCount does not increase, and the system cursor coordinates and foreground PID remain unchanged. The model cannot override these host policies via Tool parameters.

Authorize per Application, Sensitive Actions Require One-Time Confirmation¶

Access is divided into two types of leases by exact bundle ID:
- read: Read Accessibility status and requested screenshots, valid within the Session.
- control: Send UI input to the selected application, only valid for the current turn.

When no grant is configured, DSH will request approval. After the user rejects, the corresponding scope for that application in the current Session remains rejected. High-impact actions—such as external communication, sensitive data transmission, irreversible deletion, account/security/privacy changes, unsolicited installation, acceptance of legal terms, financial completion beyond explicit authorization—require calling computer_confirm before execution. The Token has a short lifespan, is one-time, and is bound to the exact app, process, observation, target handle, and action; grants cannot bypass it. Once rebinding is required for a target, the old confirmation immediately expires.

allowAllApps defaults to false. When enabled, it will ignore exact grants and grant read and control permissions to all running applications, only suitable for environments where you are fully aware of the risks.

Secure text will be displayed as [secure] in observations, and will not appear in target descriptions, tree text, Tool results, or native errors. Screenshots may still capture other visible content on the screen and need to be treated as sensitive data.

Load Skill First, Then Expose Execution Tools¶

The Bundle only contributes computer_use_activate initially. After the current Agent loads the Computer Use Skill, the execution tools will be exposed. The tools listed in the repository are as follows:

These tools do not accept AppleScript, JXA, shell, Swift, Objective-C, native selectors, arbitrary Accessibility constants, or source code.

Installation and Enablement¶

Before use, you need to meet the prerequisites listed in the repository:
- macOS 14 or newer
- DeepSeek Harness with Web or Headless Profile installed and Skill Tool mounted
- macOS Accessibility permissions for observation and native actions
- Screen Recording permission is only required when requesting screenshots
- If building from this repository, Node.js ^22.19.0 or >=24.0.0 is required

The installation command given on the community directory page is:

dsh plugin add github:Anionex/dsh-computer-use

For reproducible installations, the directory page recommends pinning the commit hash:

dsh plugin add github:Anionex/dsh-computer-use#commit

Replace #commit with the actual commit hash. The repository README additionally provides the method of installing from npm per Profile, which can be attached to both Web and Headless:

dsh plugin --profile web add @anionex/dsh-computer-use
dsh plugin --profile headless add @anionex/dsh-computer-use

dsh --profile web --dump-config | grep computer-use
dsh --profile headless --dump-config | grep computer-use

For local development, replace the package name with the absolute path of the checked-out repository. After modifying an already installed plugin, you need to restart the running dsh web host and open a new Session to allow the host to reload the Bundle and Skill catalog.

The plugin runs with the permissions of the current dsh process, and may execute code during installation. You should inspect the source code repository and license before installing.

Typical Usage¶

Load the Skill first in a new Session:

/computer-use

The introductory prompt given by the repository is:

Use Computer Use to check the running DSH Computer Use Fixture, enable the deterministic option, and report the results based on the new state returned after the action. Prioritize using Accessibility elements, do not reuse old observations.

The background fixture path in the release test can be used to understand the actual protocol:

observe exact bundle id + pid
-> element: "Targeted pointer probe", no AXPress action
-> computer_click with observationId + element index + allowCoordinateFallback
-> fresh observation
-> activation "not-requested"; pointerRouting "target-process"
-> status "Status: pointer click"

The Web Settings section will display helper integrity, Accessibility/Screen Recording status, current generation, interaction policy, restrictions, and exact application grants. Only after the user clicks will the button open the corresponding macOS privacy settings page; the plugin cannot grant TCC permissions on its own.

Accessibility and Screen Recording are UI permissions, not file system permissions. Normal usage remains under DSH workspace-write: screenshots are stored in the Session workspace, and temporary files use the Session private directory. The Bundle does not require danger-full-access. It should be noted that the danger-full-access preset uses approval/policy: never, and unauthorized applications will be blocked by the policy before the popup appears, and the plugin will return COMPUTER_PERMISSION_REQUIRED without recording this as a user rejection. In this case, you should add the exact bundle ID in Computer Use Settings, or use a preset with approval policy: ask.

The uninstall command (from README):

dsh plugin --profile web remove @anionex/dsh-computer-use
dsh plugin --profile headless remove @anionex/dsh-computer-use

After removal, Skills and Tools will be unregistered, the helper will stop working, and in-process observations and turn-level control grants will be released. Already generated screenshots and the plugin’s own computer_use_state sidecar will be retained, and can be cleaned up manually if needed.

Applicable Scenarios and Notes¶

It is suitable for these situations:
- Running DSH on macOS and needing to operate native applications without stable API/CLI
- Wanting agents to click and fill values in background windows while the user continues to use the current foreground application
- Needing to pin actions to the accessibility tree instead of replaying coordinates on expired screenshots

Situations where it is not suitable, or where a narrower interface should be used instead, are clearly stated in the repository:
- Continue using browser automation and DOM/CDP for browser tasks, which have narrower and more precise states
- When there are APIs, CLIs, or dedicated application plugins, those interfaces should still be prioritized
- OCR, visual grounding, and pixel understanding should be handed over to the standalone dsh-vision-toolkit. After loading the vision-tools Skill, pass the screenshot Artifact path to the corresponding visual tool, do not use shell to invoke tesseract, screencapture, or temporary scripts as a replacement
- Custom canvas, games, enhanced input interfaces, and future macOS versions may reject target process pointer or keyboard events; use semantic Accessibility whenever possible instead of coordinates
- Minimized, hidden, or windowless targets will fail directly; the click point must fall within a window of the selected application on some screen
- focusPolicy: activate paired with the default keyboardPolicy: activate will interrupt foreground work, and should only be used as a compatibility mode explicitly chosen by the operator
- The target application may still change its activation or focus state on its own after accepting an action

There are several security points that need to be remembered separately. The Helper is an internal DSH transmission implementation, not a public authorization API; do not treat danger-full-access as protection against direct native calls. You should use the registered Tools to retain application leases, sensitive action confirmations, and host policy checks. Custom Profiles that require interactive read grants or persistent rejection states must combine @deepseek-ai/dsh-storage-domain before this Bundle; the Web Profile already includes this dependency.

Summary¶

dsh-computer-use adds a layer of native macOS action capabilities to DeepSeek Harness: real-time observation of the accessibility tree, rejection of expired states, read/write permissions divided by bundle ID, and input paths that尽量不移动系统光标、不向全局注入指针事件. It currently only covers macOS, and the version is still the early 0.1.0. It is more suitable for developers who are already using DSH and are willing to inspect the source code before installing the plugin.

Directory page: https://deepseek-harness-plugin.com/zh-CN/plugins/dsh-computer-use/

GitHub: https://github.com/Anionex/dsh-computer-use