Preface

DeepSeek Harness (DSH) adopts a plugin-based architecture. When an agent needs to operate the desktop directly, traditional pixel-level control is risky and lacks semantic clarity. The computer-control plugin provides DSH with desktop control capabilities, including screen capture, pointer and keyboard injection, and action operations based on accessibility-tree semantics. It includes built-in safety protection mechanisms, enabling agents to safely take over desktop tasks in a controlled environment.

Plugin Overview

computer-control is a DSH plugin primarily used for desktop automation.

  • Positioning: Desktop control plugin
  • Maintainer: JohnXu22786
  • License: MIT
  • Core Value: Combines screen observation, semantic operations, and safety protection to enable controlled desktop automation.

Core Features

The plugin provides the following capabilities:

  1. Screen Capture (screen.capture): Supports full-screen or region screenshots, with PNG/JPEG output formats and optional scaling and grayscale processing.
  2. Pointer Control (pointer.*): Supports movement, clicking (left/middle/right button, single/double/triple click), dragging, and horizontal/vertical scrolling.
  3. Keyboard Control (keyboard.*): Supports single-key input, combination-key injection (independent of keyboard layout), and arbitrary Unicode text input.
  4. Semantic Actions (a11y.*): Executes activation and input operations based on accessibility-tree summaries (with skeleton/standard/full modes). Prioritizes UIA (Windows UI Access) mode and falls back to pixel-coordinate clicking on failure.
  5. Batch Execution (batch.execute): Allows multiple actions to be executed in a single call, with overall confirmation and continued execution on error.
  6. Safety Protection: Built-in emergency stop, allow/deny rules, confirmation flows, idle standby, and dry-run mode.
  7. Capability Broadcasting: Exposes tools and system status to DSH through tools.list and system.status.

Installation and Runtime

Before installing, ensure the environment meets the following requirements:

  • Python 3.9+
  • Node.js >= 18 (for bridging)
  • Windows 10/11 (full feature support)

Install the plugin:

dsh plugin --profile demo add github:JohnXu22786/computer-control

Environment check:

python -m computer_control check

List tools:

python -m computer_control list

Start the plugin service:

The plugin receives JSON-RPC 2.0 requests via stdio. The startup command is as follows:

python -m computer_control serve

Usage Example

The plugin includes a built-in demo script. It runs in dry-run mode by default and will not produce actual impacts on the real desktop:

python examples/demo.py

Security and Limitations

  • Transport Protocol: Uses line-based JSON-RPC 2.0 as the transport protocol.
  • Coordinate Mapping: The model operates against a virtual canvas (default width 1920px), and the plugin maps it back to physical pixel coordinates.
  • High-Risk Operations: Operations involving the Win key or Ctrl+Alt combinations are considered high-risk and require manual confirmation or trigger an emergency stop.
  • Platform Limitations: Full features are currently supported primarily on Windows 10/11; other platforms have limited functionality.

Links