Introduction

Browser automation tools (such as Playwright) can operate web pages well, but they cannot directly control non-browser applications. For desktop software (such as video editing tools, design software, and office suites), there has long been a lack of a way for agents to take direct control.

Plugin Purpose

dsh-desktop-automation is a macOS desktop automation plugin. It enables the dsh agent to control the desktop directly: including mouse movement, keyboard input, window management, and screenshots, and integrates a visual closed-loop workflow (screenshot → image recognition → locating → clicking → verification). The plugin is maintained by Junkrat9527 and is licensed under the MIT license.

Installation

Installing the plugin requires specifying the web profile. This can be done using one of the following three methods:

# Option 1: Direct package installation
dsh plugin --profile web add dsh-desktop-automation

# Option 2: Install from GitHub
dsh plugin --profile web add github:Junkrat9527/dsh-desktop-automation

# Option 3: Local link installation (for development/debugging)
dsh plugin --profile web add link:/path/to/dsh-desktop-automation

Core Capabilities

The plugin registers 14 agent tools covering permission checks, application/window management, input control, screenshots, and visual localization.

  1. desktop_perm: checks accessibility permissions and service status.
  2. desktop_apps: lists currently running applications.
  3. desktop_win: lists window information (including coordinates and dimensions).
  4. desktop_app: activates a specified application (the application must already be running).
  5. desktop_mouse: moves the mouse.
  6. desktop_click: performs a click (left/right click and click count are supported).
  7. desktop_drag: performs drag operations.
  8. desktop_scroll: performs scroll operations.
  9. desktop_key: simulates key presses or keyboard shortcuts.
  10. desktop_type: enters text (supports Chinese and Unicode).
  11. desktop_shot: takes a full-screen or region screenshot.
  12. desktop_see: recognizes images, describing image content or identifying UI elements (reuses dsh vision credentials).
  13. desktop_locate: visually locates elements and outputs their coordinates (type/text/box/center).
  14. desktop_stop: stops the background service.

Visual Closed Loop

By using desktop_see and desktop_locate, a complete GUI automation workflow can be implemented:

desktop_shot   → screenshot
desktop_see    → image recognition (text/UI/content description)
desktop_locate → get element coordinates
desktop_click  → click by coordinates
desktop_see    → take another screenshot to verify the result

Architecture Notes

The plugin communicates with a persistent CGEvent service over a local socket. The actual input events are emitted by the Swift service desktop-control-server. This service runs as a GUI session process to obtain macOS accessibility permissions, but it does not start automatically at startup.

Typical Usage

Check Service Status

After installation and authorization, the service status can be checked via the HTTP interface:

curl http://127.0.0.1:3080/desktop/status

Example response:

{"status":"up","perm":true,"screen":true}

Configure Vision Credentials

Before using desktop_see or desktop_locate, vision credentials (OPENCODE_GO_API_KEY) must be configured in dsh. The plugin reuses the vision capabilities of dsh-autovision, so no separate configuration is required.

Notes

  1. Accessibility permissions: Before first use, grant the plugin accessibility permissions in System Settings.
  2. Screen recording: The screenshot feature relies on the system screencapture command and is authorized by default.
  3. Service compilation: Do not recompile the server after authorization; changes to the CDHash of the ad-hoc signature may invalidate the permissions.
  4. Application startup: desktop_app only activates applications that are already running. If an application is not running, first start it with the system command open -a "Application Name".
  5. Locating model limitations: desktop_locate depends on a vision model. For complex screenshots, locating may be unstable; it is recommended to crop the target area and retry.
  6. Login auto-start: The background service is a GUI session process and does not start automatically after login; the plugin starts it on demand when needed.

Use Cases

  • Non-browser application operations: Use agents to operate desktop software such as Jianying (CapCut), Photoshop, and WPS.
  • GUI testing and acceptance: Build an automated testing loop of “screenshot → action → screenshot verification.”
  • Remote desktop orchestration: In a remote environment, use dsh to direct a local Mac to complete desktop tasks.

Summary

dsh-desktop-automation completes a missing piece of the macOS desktop control puzzle in the DeepSeek Harness plugin ecosystem. With 14 tools and a visual closed loop, it enables agents to both “see” the screen and “take action,” making it suitable for automating non-Web applications.