DeepSeek Harness (DSH) adopts an “everything is a plugin” architecture. When an agent needs to operate an Android device or verify UI changes, textual descriptions alone are insufficient to confirm results. The dsh-android-pane plugin provides DSH with a real-time Android screen panel, supporting USB physical devices and emulators, and allows both the agent and users to directly perform screen operations and verification.

Plugin Overview

The plugin is maintained by hoyyang and uses the MIT License. It pushes the scrcpy H.264 stream to the DSH web interface and provides UI element tree access, enabling the agent not only to “see” but also to “tap” and “swipe”.

Core Features

  • Real-time screen: Uses scrcpy-server to stream H.264 directly; the web interface uses WebCodecs hardware decoding, with frame rates between 30 and 60 fps. If scrcpy cannot start (for example, on older devices or when blocked by security systems), it automatically falls back to a screenshot carousel mode.
  • Agent self-verification: Includes 10 built-in context tools (tap, swipe, text input, key press, scroll, rotation, etc.). Every operation automatically returns a screenshot, and the actual changes are verified against the uiautomator UI element tree.
  • FLAG_SECURE black-screen self-repair: For black screens caused by banking, password manager, and similar apps, the plugin automatically detects the issue and provides a mitigation chain (such as adb root). If the system still blocks the operation, it falls back to a clickable UI element tree text view, with each step’s result explicitly displayed.
  • Shared human-agent screen: The panel supports tap and drag operations. While the agent is operating, the panel displays a red badge; user gestures are rejected at that time, and control ownership remains clear.
  • Session isolation: A single device is assigned to only one session at a time. If the device is idle for 10 minutes (emulator shutdown or physical device disconnection), the device is automatically released.
  • Local-only access: All video streams can be accessed only from 127.0.0.1. Requests from all other IPs return 403, preventing external access.

Installation and Enablement

Installation is zero-configuration. No API key is required, and no changes to the DSH configuration file are needed. The system must have the adb command-line tool (Android SDK) available.

Run the following command to install it:

dsh plugin add github:hoyyang/dsh-android-pane

After installation, make sure USB debugging is enabled on the device and the device is connected to your computer, or start any emulator.

Typical Usage

  1. In the DSH conversation, send the agent an instruction such as: “Connect my Android device”.
  2. The DSH web interface will show the device’s real-time screen.
  3. Ask the agent to perform an operation, for example: “Tap the Settings icon”. It will automatically perform the tap and return a screenshot for verification.
  4. You can also tap or drag directly on the screen to achieve shared human-agent interaction.

Use Cases and Cautions

  • Vendor limitations: Some heavily customized vendor systems (such as Xiaomi, OPPO, etc.) require you to manually enable “USB debugging (security settings)”. Otherwise, tapping may fail, and the panel automatically falls back to view-only mode.
  • Security and privacy: All streams are accessible only from the local machine (127.0.0.1). The dependencies scrcpy-server (Apache-2.0) and ADBKeyboard (Apache-2.0) are open-source libraries. Note that screenshots taken on the device by the agent are sent to the configured model. Please use a controlled device and a test account to avoid privacy leakage.
  • Resource usage: If multiple devices are connected at the same time, frame rates may be affected by bandwidth.

This plugin uses scrcpy technology to solve the visual verification challenge when agents operate Android devices, while maintaining security and isolation. For more details, see GitHub.