Introduction

In the DeepSeek Harness (DSH) plugin ecosystem, dsh-pc-pilot provides local Windows desktop and browser operation capabilities. This plugin gives the model a computer tool for observing and manipulating the local environment. By using the UIA accessibility tree, background synthetic input, and persistent Chromium CDP sessions, it addresses common issues in traditional automation, such as focus stealing and difficulty handling occluded windows.

Plugin Overview

dsh-pc-pilot is a computer-use runtime for Windows 10/11 that also serves as a DeepSeek Harness host plugin. It provides a command-line interface (CLI), with core capabilities including UIA accessibility access, occlusion-resilient window screenshots, background synthetic input, and persistent Chromium CDP sessions.

  • Project Name: dsh-pc-pilot
  • Maintainer: JeremyWangCY
  • License: MIT
  • Current Version: 0.4.10

Core Features

The plugin provides integrated desktop and browser control capabilities. The main features include:

Desktop Control

  • Application and Window Management: list_apps, list_windows, get_window, launch_app, get_window_state, activate_window, minimize_window.
  • Basic Interaction: click, press_key, type_text, scroll, drag, set_value, perform_secondary_action.

Browser Control

  • Session Management: browser_tabs, browser_history, browser_downloads, browser_state, browser_observe.
  • Navigation and Operations: browser_back, browser_forward, browser_wait, browser_events, browser_click, browser_type, browser_replace, browser_key.

Advanced Features

  • Identity Resolution: Supports include_text: true to obtain stable element IDs, and supports app_identity and identity_key for precise application targeting.
  • State and Recovery: Supports expect postconditions for validating postconditions, and supports the recovery: foreground_once recovery strategy.
  • Batch Execution: Supports composing and sending up to 20 ordered actions through actions, and using when as preconditions for state checks.

Runtime Mechanisms and Features

To address the focus-stealing pain point, the plugin uses Browser CDP, UIA mode, and target-window messaging to avoid grabbing focus.

  • Occlusion Resilience and Background Input: After specifying app, coordinate clicks act directly on the target window’s UIA tree or HWND, so operations can still be performed even when the window is fully covered. Background input support is routed through capability checks, and known unsupported background paths are cached to reduce retries.
  • State Binding: get_window_state returns a snapshot ID; element operations must be bound to that snapshot. Stale or incorrect snapshots are rejected, ensuring that operations are executed in the correct context.
  • Browser Session Reuse: Persistent CDP WebSocket sessions are used to support batch observation and navigation, avoiding re-establishing the connection for every operation.
  • Safety Boundaries: The plugin uses an external deadline watchdog to provide bounded failure semantics; timeouts or connection errors do not automatically trigger replays.

Installation and Enablement

No official installation command is currently provided. The installation method is to obtain the source code from the GitHub repository.

Applicable Scenarios and Cautions

  • Runtime Environment: The plugin runs with the permissions of the current dsh process. Review the source code and license before installing.
  • Session Limitations: Virtual machines, secondary desktops, and hidden Windows sessions are not supported.
  • Background Path Evaluation: Capture and background input support are evaluated per application. If an application truly requires foreground SendInput, the plugin reports that the background path is unavailable.
  • Model Ownership: Application-specific object models, automation APIs, and domain integrations should be placed in separate plugins rather than in the PC-Pilot core.

Brief Conclusion

dsh-pc-pilot provides the infrastructure for agents to operate safely in local Windows environments. For more details, refer to the plugin directory or GitHub repository.