Introduction¶

In the DeepSeek Harness (DSH) plugin ecosystem, a common path for desktop operation agents is reading the UI tree or invoking the Accessibility API. This approach relies on control structures, and differences are amplified when crossing Windows, macOS, and Linux.

beijingwahw/dsh-computer-use-plugin provides an alternative implementation: a pure vision Computer Use plugin based on DSH. It understands the interface through screenshots and performs operations such as clicking, inputting, scrolling, and dragging. Below, I introduce its positioning, capabilities, installation method, and precautions.

What is this¶

This plugin is maintained by beijingwahw and is positioned as a “pure vision Computer Use plugin built on DeepSeek Harness (DSH).”

It does not rely on the UI tree or Accessibility API and is suitable for cross-platform desktop scenarios on Win/Mac/Linux. The plugin registers tools within DSH, enabling the model to take screenshots, operate the desktop, handle popups, and verify, remember, and accumulate skills based on action results.

Core Capabilities¶

Pure Vision Grounding with Set-of-Mark Assistance¶

The plugin adopts pure vision Grounding and does not rely on the UI tree or Accessibility API.

It overlays Set-of-Mark (SoM) visual aids on screenshots, including grids, green mouse crosshairs, and element numbering boxes, to help the model make more stable coordinate judgments.

Context Management and State Anchors¶

The plugin uses context management methods such as sliding windows, converting historical images to text summaries, and llm/pre-request injection to control the size of the context.

Tool returns follow the three-part state anchor protocol {status, state_anchor, next_step}, making it easy for the model to determine the next step based on the current state.

Desktop Operations and Task Orchestration¶

The plugin supports desktop operations such as taking screenshots, clicking, inputting, scrolling, using shortcut keys, dragging, switching tabs/windows, and handling popups.

Long-running tasks can utilize the Planner-Actor dual-layer architecture of start_complex_task: first decompose the task, then execute step-by-step; fail-fast when sub-tasks fail.

Verification, Recovery, and Budget Control¶

The plugin provides multiple mechanisms surrounding visual operations:

Behavior Effect Verification: Judging if the action produces expected changes.
Two-stage Localization: More precise localization of target areas.
Change-Gated Screenshotting: Reduces repeated screenshots when the screen hasn’t changed.
Adaptive Stable Waiting: Performs verification after the interface stabilizes.
Anti-Deadlock Guard: Intercepts blind retries of actions with the same signature.
Scene Fingerprint Bonus: Improves the reliability of memory recall in the same scenario.
Dual-Scale Verification: Distinguishes between page-level and element-level changes.
Focus Tracking: Performs input and verification around the endpoint of the most recent click or drag.
Expected Anchoring: Declares expected changes before operation and verifies after.
Budget-Aware Orchestration: Aborts long tasks when they time out to avoid infinite execution.

Semantic Verification and Text Awareness¶

The plugin provides semantic verification capabilities such as find_text, read_text, and diff_view, and supports double confirmation with expected_text.

OCR is enabled on demand; language packs are downloaded online upon first use, defaulting to eng, while the Chinese scenario can use chi_sim+eng. When OCR is unavailable, semantic-related capabilities will be degraded.

Memory, Logs, and Skill Library¶

The plugin provides scenario-based UI memory, solidifying interactions where verification is effective to facilitate later recall.

It also provides action logs and replay, saving successful operation sequences as replayable actions.

The self-evolving skill library uses save_skill, match_skill, and run_skill: skills can write reliability scores based on success/failure and are used for subsequent matching.

Security Guards and Risk Gates¶

The plugin has built-in four guards: coordinate boundary validation, continuous failure circuit breaker, sensitive operation audit, and popup linkage interception.

Credential input has a dedicated gate: type_text returns ACTION_REQUIRED when reaching a sensitive focus, requiring a pause for the user to input personally; the content to be input is not echoed and is displayed as [REDACTED].

Pluggable Hybrid Mode¶

The plugin defaults to the pure vision path but also supports optional integration with local vision models and accessibility providers to obtain more precise coordinates.

Installation and Activation¶

Prerequisites¶

Requires Node.js >= 18 (recommended 22) and pnpm.

Native dependencies sharp, @nut-tree/nut-js, screenshot-desktop, and tesseract.js will be installed automatically with the plugin.

Framework dependencies are declared as peers: @deepseek-ai/cordis, @deepseek-ai/dsh-tools, @deepseek-ai/schemastery, provided by the DSH host.

Install Plugin¶

Recommended installation command:

dsh plugin add beijingwahw/dsh-computer-use-plugin --profile web

If using pnpm install:

pnpm add dsh-computer-use-plugin@github:beijingwahw/dsh-computer-use-plugin

Build artifacts are in the repository, and dist/ is distributed with the repo; no build script is executed during installation, and main points to dist/index.js. dsh.bundle points to cordis.patch.yml, and the plugin automatically registers and activates upon installation.

Start DSH¶

pnpm dsh web

Override Default Configuration¶

When you need to override default configurations, merge the insert entry from cordis.patch.yml inside the package into your own patch.

Local Development¶

git clone https://github.com/beijingwahw/dsh-computer-use-plugin
cd dsh-computer-use-plugin
pnpm install
npm run build
npm test

After modifying the source code locally, you need to regenerate dist/ and verify again.

Typical Usage¶

Desktop Operations¶

After installing and starting DSH, you can use capabilities such as taking screenshots, clicking, inputting, scrolling, using shortcut keys, dragging, switching tabs/windows, and handling popups in tasks. Tool returns {status, state_anchor, next_step}, allowing the model to continue execution or correct based on this.

OCR and Semantic Verification¶

When OCR is enabled, you can combine find_text, read_text, diff_view, and expected_text for semantic verification and visual difference detection.

Enable example:

enableOcr: true

If OCR is unavailable, semantic-related capabilities will be degraded.

Long Tasks¶

Use start_complex_task to submit long-running tasks. The plugin will decompose steps and execute them step-by-step, fail-fast when sub-tasks fail, and control the execution scope in conjunction with budget-aware orchestration.

Skill Solidification¶

Use save_skill to solidify skills, use match_skill to match similar skills, and then use run_skill to execute them. The reliability of each execution result is written back and used for subsequent matching.

Sensitive Input¶

When the input target is marked as a sensitive focus, type_text returns ACTION_REQUIRED. At this point, automatic input should be paused, and the user should be asked to input personally; the content to be input is displayed as [REDACTED] and does not echo plain text.

Use Cases and Notes¶

Suitable for:

Cross-platform desktop operations on Windows, macOS, and Linux.
Visual automation scenarios that do not rely on the UI tree or Accessibility API.
Processes requiring action verification, failure recovery, log replay, and skill solidification.
Plugin development and local debugging within DSH.

Notes:

The plugin executes desktop operations based on the permissions possessed by the current dsh process. It is recommended to check the source code, dependencies, and license before installation.
The materials do not clearly specify the license name; the files declaration in package.json contains LICENSE, and the authorization scope should be confirmed by yourself before use.
OCR language packs need to be downloaded online upon first use; offline environments need to accept degraded semantic capabilities.
The DSH plugin ecosystem emphasizes “everything is a plugin”. The community directory is an independent site and does not represent the official app store of DeepSeek or Hypersphere.

Links¶

GitHub:

https://github.com/beijingwahw/dsh-computer-use-plugin

Directory Page (Resource Clues):

https://www.skillhub.cn/plugins/beijingwahw/dsh-computer-use-plugin

beijingwahw/dsh-computer-use-plugin: DSH-based pure vision Computer Use plugin