Introduction

DeepSeek Harness (DSH) adopts a plugin-based architecture, allowing core functionality to be extended through additional components. dsh-vdesktop is one such plugin for solving GUI application interaction problems. It uses the Windows CreateDesktopW API to create an invisible Win32 virtual desktop, running GUI applications in an isolated environment. DSH uses a built-in HTTP frame-capturing service and a JSON-RPC toolchain to enable LLM Agents to operate these applications, while also providing users with a real Web panel for monitoring and takeover.

Plugin Overview

This plugin is maintained by developer jafewff and licensed under the MIT License. It is a plugin based on the cordis mechanism, with its core value providing a complete set of virtual desktop lifecycle management tools and a visual interaction panel, bridging the gap between pure command-line agents and GUI applications.

Core Features

The plugin mainly includes the following five capabilities:

  1. Virtual Desktop Lifecycle Management
    Supports creating, attaching, listing, and closing virtual desktops. The plugin automatically cleans up “orphan” desktops with the dshv- prefix to prevent resource leaks.

  2. Agent Toolchain
    Provides 12 vdesk_* tools, covering desktop operations, full-screen/region screenshots, mouse operations (click/drag/scroll), keyboard input (including Chinese and emoji), text reading/writing, and file access.

  3. Frame Data Service
    Includes a built-in frameServer (HTTP service). Ports are automatically probed from 8790 to 8799 for low-frequency polling of PNG frame data by the Web panel. Response headers include cursor coordinates (X-Cursor-X/Y), which are overlaid by the frontend panel rather than burned into the image.

  4. Semantic Clicking
    The panel and Agent share the same execution layer. For standard windows, title bar buttons are identified through WM_NCHITTEST; for custom-drawn windows, operations go through postmessage. Clicking taskbar buttons is equivalent to window activation or launching.

  5. Process Supervision
    The Python Sidecar is hosted as a subprocess, with heartbeat monitoring, lazy restart, and idle reclamation mechanisms to ensure service stability.

Installation and Enablement

Installation requires manually building or linking the repository and configuring the Sidecar path.

  1. Obtain and Build the Code
    Clone the repository and run the build commands. The repository includes build artifacts in lib/, so rebuilding is usually unnecessary unless the source code has been modified.
    git clone https://github.com/jafewff/dsh-vdesktop dsh-vdesktop
    cd dsh-vdesktop
    npm install
    npm run build
  1. Install the DSH Profile
    Add the dependency to package.json in the DSH Web Profile directory (for example, C:\Users\<you>\.dsh\profiles\web).
    "dependencies": {
      "dsh-vdesktop": "link:<本仓库绝对路径>"
    }

Add the plugin to `dsh.profile.bundles` in the profile configuration, then run `pnpm install` and restart DSH.

  1. Configure the Sidecar Path
    The plugin launches the Sidecar using python -m dshv --rpc stdio. Update sidecarCwd in cordis.patch.yml or the configuration file to point to the sidecar directory in the repository (absolute path).

  2. Post-Installation Self-Check
    After restarting DSH, verify that the service is ready using curl.

    # Verify the desktop list endpoint
    curl.exe -s http://127.0.0.1:8790/vdesk/desktops

    # Verify the frame capture endpoint
    curl.exe -s -o NUL -w "%{http_code}" "http://127.0.0.1:8790/vdesk/frame?desktop=dshv-e2e"

Open the DSH Web GUI and confirm that `VDesktopPanel` appears and the screen refreshes normally.

Usage Notes

Web Panel Operations

Use VDesktopPanel in the DSH Web GUI. Users can select or create a virtual desktop, and the screen refreshes using low-frequency polling. Clicking the title bar (×/minimize/maximize) or taskbar buttons in the view maps directly to real window management messages.

Agent Tool Calls

Agents can interact with the virtual desktop through tool calls.

  • Create a Desktop and Launch an Application: Use vdesk_desktop_create. Note that launch is asynchronous: it immediately returns a PID but windowReady is false, so the top-level window must be matched by PID.
  • Input Operations: Because the virtual desktop is not in the foreground, SendInput is rejected, and client-area clicks actually go through postmessage. Keyboard tools support Chinese and emoji.
  • Verify Installation: You can use vdesk_desktop_list to view the desktop list.

Behavioral Details and Cautions

  • Input Fallback Chain: Client-area clicks go through postmessage, while standard window title bars use WM_NCHITTEST. The Start menu is a “best-effort” implementation, and full operation may still require a real input channel.
  • Asynchronous Launch: Application startup has a delay. Agent scripts should wait for the window to appear before performing operations to avoid invalid clicks caused by PID matching failure.
  • Cursor Management: Cursor positions are maintained by the Sidecar and are synchronized after each clicked frame.
  • Ports and CORS: If the FrameServer port is in conflict, it automatically increments. The CORS whitelist defaults to http://127.0.0.1:3080.
  • DPI Scaling: The plugin has been tested on Windows 10/11 at 100% scaling (source 1920x1080, frame 1280x720).

Summary

dsh-vdesktop provides DSH with the ability to run GUI applications in isolated virtual desktops. It enables Agents to read/write text, take screenshots, and control the mouse and keyboard through 12 tools, while its HTTP service and Web panel address the needs for visualization and human takeover. During installation, pay attention to the Sidecar path configuration and the asynchronous launch handling logic.