Introduction¶

The difficulty in enabling agents to operate on the Windows desktop does not lie in “clicking once,” but in three key issues: the screenshot captured by the model might be of the wrong window; custom-drawn interfaces like WeChat or Electron lack an accessibility tree, making the element_index path unusable; operations like deletion or payment lack confirmation steps, meaning there is no going back if an error occurs. Common approaches either rely entirely on guessing coordinates from screenshots or require the target application to provide a full UI Automation tree. Neither is stable on a real desktop.

Below, we introduce dsh-computer-use-windows, a plugin designed for DeepSeek Harness (DSH). DSH’s philosophy is “everything is a plugin,” and this plugin fills the gap in Windows Computer Use: window discovery, UI Automation element operations, occlusion-aware screenshots, 3x3 grid visual positioning, and risk operation approval.

What is This¶

Maintained by nanbbb, dsh-computer-use-windows is currently at version 0.2.1 under the MIT license. In a nutshell, it provides Windows Computer Use tools with security gatekeeping for DeepSeek Harness, compatible with ModLens for visual positioning.

Before use, there are several prerequisites:

• It is currently targeted at Windows 10/11 x64 (MVP stage);
• Node version requirement is >= 22.19 (from the engines field in package.json);
• The native helper comes with a built-in .NET runtime and is fully self-contained, requiring no separate installation;
• @liustack/modlens must remain enabled, as this plugin does not duplicate or replace its visual provider.

Core Features¶

Window Discovery and Foreground Input¶

computer_list_windows performs Windows window discovery. Before operations, the plugin activates the target window to the foreground, then executes clicks, typing, and key presses.

UI Automation and Occlusion-Aware Screenshots¶

Standard controls use UI Automation: the plugin returns element discovery results and focus status, and operations bind to element_index. Window screenshots are occlusion-aware: when the window is blank or obscured, the native helper will reactivate the window and retry; if still obscured, it reports a clear error instead of returning a screenshot of the erroneous window.

ModLens-Compatible 3x3 Visual Positioning¶

computer_observe returns a grid_screenshot_path with annotations A1-C3. For custom-drawn UI without an accessibility tree, use modlens_read_image to read the grid, select the target cell among A1-C3, then use computer_zoom to zoom in gradually, repeating until the area is small enough to click.

Operations Bound to Observation Snapshots¶

Both element and coordinate operations are bound to observation snapshots: element_index and coordinates are only available within the validity period of the snapshot; once the snapshot expires, they become invalid, preventing operations on outdated interfaces.

Approval and Tray Wake-up¶

Risk operations are confirmed via a one-time approval prompt from DeepSeek Harness. Tray wake-up is handled by computer_hotkey (e.g., ctrl+alt+w) and computer_tray.

Installation and Enablement¶

Install from npm:

npx -y @deepseek-ai/dsh plugin --profile web add dsh-computer-use-windows

Or install from GitHub:

npx -y @deepseek-ai/dsh plugin --profile web add github:nanbbb/dsh-computer-use-windows

After installation, you must restart DeepSeek Harness for it to take effect. To build from source, run verification and packaging first, then install the generated tgz into DSH:

npm run verify
npm pack
npx -y @deepseek-ai/dsh plugin --profile web add C:\absolute\path\to\dsh-computer-use-windows-0.2.1.tgz

Similarly, restart DeepSeek Harness after installation.

Configuration¶

Configuration is written in the config field of cordis.patch.yml. All fields are optional and have default values:

allowedExternalLabels only allows external type send/submit labels; when matched, they are processed as an explicit model risk; delete, payment, and login operations are not within the allowance scope and are always escalated to approval.

Typical Usage¶

Calculator Task¶

Example task: Use Computer Use to open an existing calculator window and calculate 19 × 7, without operating on other applications. The process is:

1. computer_list_windows to select the only returned window;
2. computer_observe to get an observation snapshot;
3. Execute an operation;
4. computer_observe again to confirm the result.

Custom-Drawn UI Without Accessibility Tree¶

Custom-drawn interfaces like WeChat and Electron cannot get element_index, so use the screenshot path:

1. computer_observe to get grid_screenshot_path;
2. modlens_read_image to read the grid and select the target cell among A1-C3;
3. computer_zoom to zoom into that area, repeating until the area is small enough, then click.

ModLens may time out or be imprecise: retry if unreadable and cross-verify with computer_zoom; do not rely solely on a single grid result. Standard controls should still use the element_index path of UI Automation.

Waking WeChat in the Tray¶

WeChat is hidden to the tray rather than minimized, so wake it up before observing:

computer_hotkey: ctrl+alt+w

Or use computer_tray. Do not rely on the helper to reveal a blank window.

Applicable Scenarios and Notes¶

Suitable scenarios: Developers who need agents to operate on real desktop applications (mixed standard controls and custom-drawn UI) on Windows 10/11 x64. During the MVP stage, the platform scope is limited to Windows 10/11 x64.

Before use, there are a few things that must be clear:

1. This is a foreground mode. The target window is activated to the foreground before operations, and the physical mouse is moved to the operation point; it cannot run in the background. Do not use this machine while the agent is working.
2. The security model is fail-closed. Risk calls use the Harness approval service and fail immediately if approval support is missing; instructions appearing on the screen never count as user authorization.
3. The native helper will reject a batch of requests: invisible windows, invalid handles, coordinates outside the window, password controls, terminal windows, authentication UI, and locked desktops.
4. The plugin runs with the permissions of the current dsh process; what it can do depends on the environment where you launched DSH. It is recommended to check the source code and license before installation; this plugin is MIT-licensed.

Conclusion¶

In summary: dsh-computer-use-windows combines window discovery, UI Automation, occlusion-aware screenshots, and 3x3 grid visual positioning, uses observation snapshots to constrain operations, and uses the approval service as a safety net for risks, completing the Windows Computer Use piece within the DSH plugin ecosystem. With the installation, configuration, and example workflows provided above, you can try it directly in your own workflow.

• Community Plugin Directory: https://www.skillhub.cn/plugins/nanbbb/dsh-computer-use-windows
• GitHub Repository: https://github.com/nanbbb/dsh-computer-use-windows

The directory page is an independent community site and has no official affiliation with DeepSeek or High-Fin.