Introduction

Under DSH’s plugin mechanism, tool capabilities can be augmented for agents. Two common requirements exist: first, passing local screenshots or images to vision models for understanding; and second, enabling agents to perform real clicks, swipes, text input, and key verification within Android applications.

dsh-vision-android is a DeepSeek Harness plugin written around these two needs: on one side, sending local image files to vision models to ask questions, and on the other, executing Android UI operations via adb.

What is this

dsh-vision-android is a DeepSeek Harness plugin designed for “Multimodal Vision + Android UI Automation”.

It provides 11 tools, covering image questioning, screenshots, screen understanding, UI hierarchy scraping, real clicks, swipes, text input, key presses, APK installation, app launching, and screen size querying. The project license is MIT, and the GitHub repository is https://github.com/superclaude1/dsh-vision-android.

Core Features

The following explains features grouped by capability.

Visual Understanding

  • vision_ask: Sends local image files to the vision model to ask questions.
  • mobile_look: Takes a screenshot and performs visual understanding, returning screen description and coordinates of interactive elements.
  • mobile_screenshot: Takes a screenshot and saves it as PNG, returning the file path.
  • mobile_screen_size: Retrieves the physical screen resolution.

Android UI Automation

  • mobile_dump: Scrapes UI hierarchy coordinates via uiautomator.
  • mobile_tap: Performs a real click at specified coordinates via adb.
  • mobile_swipe: Executes real swipe gestures via adb.
  • mobile_text: Inputs ASCII text; spaces are automatically escaped.
  • mobile_keyevent: Sends key events such as back, home, enter, tab, esc, dpad_*, etc.
  • mobile_install: Installs APK using adb install -r.
  • mobile_launch: Launches components via am start, or launches packages using monkey.

Installation and Activation

The command to install into the web profile is:

dsh plugin --profile web add ./dsh-vision-android

When activating, it is recommended to place sensitive machine configurations in $DSH_HOME/cordis.patch.yml, or within the profile’s own patch file.

visionApiKey is recommended to be set via environment variables rather than in configuration; do not commit keys to the repository.

Typical Usage

A reproducible model conversation example is:

Please open the book "Lord of the Mysteries": first use mobile_look to see the current interface, find the coordinates of this book, use mobile_tap to click, and then use mobile_look to confirm entering the table of contents page.

The execution order can be understood as:

  1. Call mobile_look to obtain the current interface description and element coordinates.
  2. Call mobile_tap to click based on the returned coordinates.
  3. Call mobile_look again to verify entry into the table of contents page.

This forms a closed loop of “Screenshot → Visual Understanding → Real Click → Screenshot Verification”.

If a coordinate is critical, you can first use mobile_dump to scrape UI hierarchy coordinates, then cross-verify them with the coordinates returned by vision.

Applicable Scenarios and Notes

It is suitable for two types of work: requiring agents to read screenshots or images; and performing UI operation verification within Android applications.

Notes before use:

  • The plugin runs with the permissions of the current dsh process. It is recommended to check the source code and license before installation.
  • The license is MIT.
  • mobile_text only supports ASCII; Chinese input requires the device’s IME; spaces are automatically escaped to %s.
  • Visual coordinates are pixel values; coordinates from mobile_dump are always precise, while visual coordinates may occasionally deviate. You can use mobile_dump to verify before critical clicks.
  • The vision model may misinterpret non-app UI elements, such as the system soft keyboard, as application elements. Cross-verify with mobile_dump before critical operations.
  • visionApiKey is recommended to be set via environment variables; do not commit keys to the repository.

Conclusion

The value of dsh-vision-android lies in connecting “looking at the screen” and “operating the device” into DSH’s agent workflow: first letting the model understand the interface, then using adb to execute operations, and finally taking a screenshot again for verification.

GitHub Repository: https://github.com/superclaude1/dsh-vision-android.

The URL for the community directory page is not provided in this material, but you can search for the plugin name dsh-vision-android in the community directory.