Introduction

The core philosophy of DeepSeek Harness (DSH) is “everything is a plugin.” In the existing DSH ecosystem, macOS already has a mature computer-use plugin, but Windows has long lacked a native implementation. dsh-computer-use-win is designed to fill this gap. It allows DSH plugins to control the Windows desktop environment through screenshots, UI Automation element discovery, OCR, and authorized input, enabling text models to interact with Windows applications via accessibility.

Installation and Enablement

This plugin must be installed on a Windows system and depends on an installed DSH CLI.

  1. Run the install command:
    dsh plugin --profile web add https://github.com/secretxuan/dsh-computer-use-win/releases/download/v0.7.0/dsh-computer-use-win-0.7.0.tgz
  1. Verify that the installation is effective:
    dsh --profile web --dump-config | Select-String computer-use
  1. Start DSH:
    dsh --profile web

Core Capabilities

The plugin exposes Windows operating capabilities through a series of tools, covering screenshots, observation, discovery, and interaction:

  • View and Discovery

    • computer_screenshot: Captures the full screen or a specific window. Vision routes receive the image; text-only routes receive the saved PNG path.
    • computer_list_windows: Lists visible top-level windows (process, PID, title, screen rectangle, activation state).
    • computer_list_elements: Uses UI Automation to search for controls and returns names, types, physical pixel rectangles, and click coordinates.
    • computer_ocr_image / computer_ocr_window: Runs Windows OCR on an image or window and returns text boxes with absolute coordinates.
    • computer_observe_image / computer_observe_window: Converts an existing image or window into a generic text-model UI mapping (OCR, rough regions, click centers, and visual candidates).
    • computer_observe_media_controls: Reports generic media control states, such as play/pause status, without custom pixel scripts.

  • Interaction

    • computer_click / computer_move_mouse: Click or move using screen coordinates.
    • computer_type_text: Inputs text through clipboard paste (reliably handling CJK), with SendInput as a fallback.
    • computer_press_key / computer_scroll / computer_drag: Key, scroll, and drag operations.
    • computer_open_settings: Opens Windows Settings via the ms-settings: URI.
    • computer_search_in_window: Performs a generic search flow (locate the field, enter input, and press Enter).
    • computer_locate_text / computer_click_text: Locates and clicks visible text using OCR.

Configuration

After installation, parameters usually need to be adjusted in the DSH configuration file. Complete the computer-use-win configuration in cordis.patch.yml:

- id: computer-use-win
  config:
    requireApproval: true
    maxScreenshotBytes: 8388608

Note: requireApproval: false is only suitable for unattended testing or temporary environments.

Typical Usage

Choose the tool combination based on the model type (vision or text-only).

Recommended workflow for text models:
1. computer_list_windows
2. computer_list_elements(window: "target app")
3. If UIA diagnostics indicate limited coverage, use computer_observe_window(window: "target app")
4. Use suggested actions directly when they match the task
5. Use computer_observe_media_controls to check play/pause status instead of custom pixel scripts
6. Use computer_search_in_window(window: "target app", query: "...") to search
7. Use computer_locate_text or computer_click_text to locate text
8. For icon-only targets, use computer_click with high-confidence visual candidates
9. Use computer_type_text or computer_press_key to input
10. Verify using computer_list_windows, computer_list_elements, computer_observe_window, or OCR

Recommended workflow for vision models:
In DSH, select DeepSeek-V4-Flash-Vision-Exp so the model inspects actual screenshots instead of OCR summaries:
1. computer_screenshot(window: "target app")
2. Inspect the returned image

Notes

  1. Global input injection: Windows input injection is global. Clicks occur at whatever location is under the coordinates, not only in the currently focused window.
  2. Privilege issues: Elevated administrator windows may ignore input from non-elevated plugin processes.
  3. VDI filtering: Win-key combinations use SendInput, and some VDI products filter this feature. Use semantic tools wherever possible, such as computer_open_settings.
  4. UIA dependency: UI Automation coverage depends on whether the target application exposes useful accessibility metadata. For CEF/Chromium/custom-rendered applications, use computer_observe_window, computer_search_in_window, or computer_click_text.
  5. Text-only limitation: Text-only DeepSeek routes cannot directly inspect screenshot pixels. computer_screenshot returns the saved PNG path on these routes. Use observation/OCR tools to convert the path or window into text, UI regions, and coordinates.
  6. Observation is not a model: Visual observation is not a vision model. Icon/button candidates are heuristics with confidence and evidence; verify after clicking.

Conclusion

dsh-computer-use-win provides DSH with the necessary toolset for interacting with Windows interfaces. By relying on system accessibility and OCR processing, it addresses the issue that text models cannot directly inspect pixels. Please check the plugin repository for source code and updates.

  • GitHub: https://github.com/secretxuan/dsh-computer-use-win
  • SkillHub Catalog: https://www.skillhub.cn/plugins/secretxuan/dsh-computer-use-win