Introduction

The DeepSeek Harness (DSH) ecosystem emphasizes pluginization. When using text-only models for desktop automation, users often face the challenge of lacking visual capabilities or relying on external APIs. computer-user solves this problem by allowing text-only models to “see” the screen through local image analysis tools and directly control the operating system to perform operations.

What It Is

computer-user is a DSH plugin that supports Codex-style computer operations: reading the screen and driving the mouse and keyboard via Win32 SendInput. It enables text-only models to work with local image analysis tools (such as picturereader) to “see” the screen and execute operations, running entirely locally with no external API calls.

Core Features

  • Screenshot: Captures the entire virtual screen as a PNG, supporting multiple monitors and DPI awareness.
  • Mouse and keyboard control: Drives mouse movement, clicks, double-clicks, drag-and-drop, and wheel scrolling via Win32 SendInput.
  • Text input: Supports inputting arbitrary UTF-16 text, including CJK characters.
  • Combination key support: Supports keyboard combinations, such as ["ctrl","c"].
  • Text-only model support: With picturereader, any text-only LLM can drive desktop operations.
  • Localization: Screenshot capture, analysis, and input are all performed locally, with no external API calls.

Installation and Activation

In a DSH profile, run the following command to install it:

dsh plugin --profile web add computer-user

After installation, restart DSH or manage the screen plugin in EAC settings for the tool to become active in any session.

Typical Usage

  1. Screenshot and analysis: Call computer_screenshot to capture the screen, then pass the returned path to image_scan or image_ocr in picturereader.
  2. Text input: Use computer_type to enter arbitrary UTF-16 text.
  3. Clicking and interaction: Call computer_click at specific coordinates, or execute computer_keypress to trigger a shortcut.
  4. Scrolling and drag-and-drop: Use computer_scroll to scroll a page, or use computer_drag to perform a drag operation.

Applicable Scenarios and Notes

  • Applicable scenarios: Suitable for local, privacy-safe desktop automation workflows, especially when using text-only models such as DeepSeek V4 Flash.
  • Coordinate system: Coordinates are relative to the origin of the virtual screen, which combines all monitors, rather than relative to a single monitor.
  • Target window: Before performing operations, first locate the target window to avoid background icons interfering with OCR and clicks.
  • Output Guard: Output Guard is enabled by default to prevent the model from writing tool calls as ordinary conversational text. If you need to output code snippets, you must disable it in advanced settings.
  • Compatibility: Supports Headless mode and is compatible with DeepSeek Harness EAC.

Brief Conclusion

computer-user completes the local closed loop of “seeing, acting, and verifying,” filling the gap in desktop operations for text-only models. The open-source repository is as follows:

https://github.com/jing-hy/computer-user