Preface

When performing agent tasks in DSH, controlling Android devices often encounters several specific issues: whether the device has completed debugging authorization, which interactive elements are currently on the screen, whether the actions issued by the model have taken effect, and which actions must be approved manually.

dsh-mobile-gui-agent is a DeepSeek Harness plugin. It controls Android devices via ADB, integrating screenshots, UIAutomator hierarchy, action verification, approval, and a web mobile view into a single plugin. This enables DSH to handle mobile tasks in an “observe-decide-act-verify” loop.

What This Is

kunjinkao-os/dsh-mobile-gui-agent is an Android Mobile GUI Agent plugin for DeepSeek Harness. It provides ADB control, verified iterative actions, approval, and a web mobile view.

The project is maintained by kunjinkao-os under the MIT license. The currently installable version is v0.2.1.

Core Capabilities

Below are the verified capabilities provided by the plugin.

  • ADB Control: Device discovery, wireless connection, screenshots, UIAutomator hierarchy capture, tap, long press, swipe, text input/replacement, key events, back, home, and package launching.
  • Observation Input: Screenshots and pruned semantic UI observations; element IDs are locally valid within the current observation.
  • Harness Tools: Provides strict phone_observe and phone_act tools.
  • Action Loop: Executes one meaningful action per model turn, then obtains a new observation and performs deterministic verification.
  • Stability Protections: Stale-element protection, adaptive screen stabilization, stuck detection, step and time limits, and recoverable ADB errors.
  • Approval: Triggers Harness approval for semantic controls that may involve sending, posting, deleting, purchasing, paying, transferring funds, dialing, installing, or modifying account security.
  • Web View: Provides a blank-session mobile_gui_agent launcher and conversation tab, supporting device selection, wireless connection, screenshot refresh, task control, action overlays, and verified steps.
  • Testing Capabilities: Provides a fake device and scripted state transitions for keyless Agent-loop tests.

Requirements

Before using this plugin, ensure your environment meets the following conditions.

  • Requires the DeepSeek Harness Web profile. The mobile_gui_agent entry is a browser client contribution, so it won’t appear in headless-only profiles.
  • Requires an Android physical device or emulator visible to adb devices.
  • Android device must have Developer options and USB debugging enabled, with RSA debugging authorization completed.
  • Node.js requirement: ^22.19.0 || >=24.0.0.
  • DeepSeek Harness compatibility: ^0.1.0-rc.5; verified against upstream commit 47f943859bef60e4160492346772ded9b24f765a and built/tested on published 0.1.0-rc.6 Harness packages.

Installation and Enabling

First, confirm the Android device is available. The following command lists connected devices:

adb devices -l

The device must be authorized for subsequent plugin operations via ADB.

Next, install the pinned version to the Harness Web profile:

dsh plugin --profile web add github:kunjinkao-os/dsh-mobile-gui-agent#v0.2.1

This plugin can control real devices, so it’s recommended to pin a reviewed tag or commit during installation.

After installation, check the configuration and start the Web profile:

dsh --profile web --dump-config
dsh --profile web

Following these steps, the mobile_gui_agent entry will be available in the web interface.

Typical Usage

It is recommended to use the following workflow to separate regular Harness session messages from mobile tasks.

  1. Select a workspace and send a brief initialization message in a regular Harness conversation, e.g.:
   Prepare a mobile task

  1. Select the mobile_gui_agent conversation tab.

  2. Choose the connected device and enter the actual mobile command only in the Task field of mobile_gui_agent.

  3. Select Start, then follow the verified steps to view execution results.

Alternatively, use the blank-session mobile_gui_agent launcher to open the task panel. The above workflow helps maintain a clear boundary between regular Harness sessions and mobile tasks.

Input and Permission Boundaries

Unicode input uses an external ADB Keyboard helper:

com.android.adbkeyboard/.AdbIME

If this helper is missing, you can configure an absolute host path to a reviewed APK via adb.unicodeImeApkPath. The plugin itself does not download the APK; if APK installation is involved, explicit Harness approval is still required.

The plugin does not install Android accessibility services.

For custom canvas, WebView, game, and image-only controls, they may appear in screenshots but not in the UIAutomator hierarchy. For such screens, use a vision-capable model input or the optional PhoneVisionProvider.

Use Cases and Considerations

This plugin is suitable for developers or users who want to add Android device control capabilities to DSH, especially in scenarios involving viewing device status in the web interface, executing mobile tasks, and approving sensitive operations.

Before use, note:

  • It calls local capabilities like ADB via the current Harness/dsh process and performs operations on Android devices.
  • Before installation, check the source code, license, and version pinning.
  • Requires the DeepSeek Harness Web profile; headless-only profiles won’t show the mobile_gui_agent entry.
  • Actions involving sending, posting, deleting, purchasing, paying, transferring funds, dialing, installing, or modifying account security require Harness approval.
  • When controlling real devices, install a pinned tag or commit rather than using unreviewed versions arbitrarily.

Conclusion

The value of dsh-mobile-gui-agent lies in integrating Android GUI control into the DSH plugin ecosystem: ADB actions, screen observation, action verification, approval, and web view are all part of a unified workflow.

Project URL:

https://github.com/kunjinkao-os/dsh-mobile-gui-agent