Introduction

The core design philosophy of DeepSeek Harness (DSH) is “everything is a plugin.” When developing computer-use agents in a Linux environment, existing solutions often rely on Node native modules, PowerShell, or accessibility frameworks, which increases the complexity of environment configuration. As a plugin for Linux X11, dsh-computer-use addresses the requirements for screenshots and real mouse and keyboard input through a lightweight Python XTest helper.

Plugin Purpose

This is a DeepSeek Harness plugin that allows agents to take screenshots, move the mouse, click, drag, scroll, and type keyboard input in a Linux X11 desktop environment. It does not depend on Node.js build steps or accessibility frameworks, and instead calls libX11 and libXtst directly through Python ctypes. The plugin is maintained by asdasdsdsdasdasdasd, uses the MIT license, and is not an officially maintained DeepSeek package.

Core Features

The plugin provides a set of tool interfaces for controlling the desktop environment:

  • computer_screenshot: Captures the screen (full screen or region) and returns the image plus scaling metadata.
  • computer_move_mouse: Moves the pointer to the specified pixel coordinates.
  • computer_click: Moves and clicks the mouse.
  • computer_drag: Drags between two points.
  • computer_scroll: Scrolls the wheel at the specified location.
  • computer_type: Types text into the focused window.
  • computer_key: Sends a key press or key combination.
  • computer_cursor: Reads the current pointer position.
  • computer_screen_info: Retrieves screen geometry information and XTest availability.

Installation and Enablement

Add the plugin to an existing Harness configuration. Make sure your environment has python3, libX11, libXtst, and ImageMagick installed.

dsh plugin --profile web add github:asdasdsdsdasdasdasd/dsh-computer-use

After installation, restart the DSH process and use dsh --profile web --dump-config to confirm that the configuration contains the # == dsh-computer-use layer.

Configuration Notes

The plugin uses danger-full-access sandbox mode by default. Because real input must be performed through XTest, it is recommended to use it in a controlled Linux environment.

You can adjust the configuration in cordis.patch.yml:

- id: computer-use
  config:
    helper: /absolute/path/to/xc.py   # 默认为 index.js 旁边的 xc.py
    shotDir: /tmp/dsh-computer-use    # 截图临时目录
    defaultWidth: 1280                # 图片宽度上限
    sandboxMode: danger-full-access   # 沙箱模式
    observeAfterAction: true          # 操作后附带截图

Usage Example

The plugin provides a Python helper script xc.py for validating the environment. The following command should return JSON containing the screen size and xtest status:

python3 xc.py info

Coordinate calculations are based on native X11 pixels. If the screenshot has been scaled, use the following formula to convert:

screen_xy = region_xy + image_xy * scale_xy

Limitations and Notes

  • Platform limitation: Supports Linux X11 only. Wayland, macOS, and Windows are not supported.
  • Dependencies: Requires python3, libX11, libXtst, and ImageMagick. No Node build step, no PowerShell, and no accessibility framework.
  • Permissions: The default sandboxMode is danger-full-access; the plugin has full desktop control, including password entry and destructive operations.
  • Security: Do not enable this plugin in shared sessions or on machines containing sensitive information. Treat it as a full desktop control tool.

Summary

If your agent runs in a Linux X11 environment and you want to avoid complex Node or accessibility dependencies, dsh-computer-use provides a direct X11 screenshot and input solution. After installation, the agent can interact with the real desktop through the computer_* tool family.

Related resources:
* Plugin catalog: https://www.skillhub.cn/plugins/asdasdsdsdasdasdasd/dsh-computer-use
* GitHub repository: https://github.com/asdasdsdsdasdasdasd/dsh-computer-use