Preface¶

The philosophy of DSH is “everything is a plugin,” and mobile control capabilities can also be plugged into the agent’s toolset. dsh-mobile-control is such a DSH plugin: it drives iOS simulators via WebDriverAgent (WDA), providing operations such as screenshots, finding UI elements, clicking, opening URLs, and exporting the accessibility tree. Unlike approaches that rely on macOS accessibility or idb, this plugin uses WDA’s HTTP service. After installation, it registers 8 mobile_* DSH tools to the agent, allowing iOS simulator operations to be completed directly in the session using natural language or tool calls.

What is this¶

dsh-mobile-control is a DeepSeek Harness (DSH) plugin, positioned as:

DeepSeek Harness plugin · Control mobile devices / DSH plugin for controlling mobile devices

The repository is maintained by PangYiMing under the MIT license.

It solves the core problem of: allowing DSH agents to operate iOS simulators through a set of mobile_* tools. The plugin relies on WDA at its core, with capabilities including screenshots, element finding, clicking, opening URLs or custom schemes, exporting the accessibility tree XML of the current foreground app, and using simctl privacy grant for privacy pre-authorization.

Core Features¶

Below are the verified main capabilities:

Drive iOS simulators via WebDriverAgent (WDA)
WDA exposes a localhost:8100 HTTP service
Port adaptive; the real port may be written to ~/.cache/webdriveragent/wda.port
iOS simulator screenshots
Find UI elements by accessibility label / class
Click elements, or click by logical coordinate points
Open URLs or custom scheme deep links
Export the accessibility tree XML of the current foreground app
Use simctl privacy grant for pre-authorization to avoid system popups blocking automation
Idempotent WDA startup and port adaptation
Register 8 mobile_* DSH tools after plugin installation

WDA’s characteristics are: zero macOS accessibility authorization, no dependency on idb, and the ability to click any app, including system popups.

Installation and Enablement¶

First, confirm the runtime environment. package.json declares:

Node >=20.11
peerDependencies are @deepseek-ai/dsh-tools and @deepseek-ai/cordis

Then install from GitHub:

dsh plugin --profile demo add github:PangYiMing/dsh-mobile-control

Installing from GitHub source requires the prepare build. After installation, the plugin registers 8 mobile_* DSH tools for the agent to call in sessions.

Typical Usage¶

1. Start WDA¶

First, start WDA:

bash scripts/wda-up.sh [UDID]  # Use current booted simulator if UDID is not passed

The script uses idempotent startup: if WDA is already running, it returns directly; if there is a cache, compilation can be skipped; if there is no cache, cloning and compiling must be completed first. When adapting the port, the real port may be written to:

~/.cache/webdriveragent/wda.port

Both the plugin and the script read this port instead of assuming port 8100.

2. Check WDA Status¶

By default, you can check localhost:8100/status:

curl -s localhost:8100/status | head -c 200

If the actual port after port adaptation is not 8100, you should replace 8100 with the port read from ~/.cache/webdriveragent/wda.port.

3. Call in Agent Session¶

After installing the plugin, you can trigger tools directly in the agent session using natural language. For example:

Use mobile_wda_up to start WDA, then mobile_tap to click (268, 492)

Below are some verified usage examples for tools:

Use mobile_find_element to find UI elements by accessibility label / class, returning the element ID
Use mobile_tap to click an element (elementId) or by logical coordinate points (x / y)
Use mobile_open_url to open URLs or custom scheme deep links
Use mobile_grant_privacy for pre-authorization to avoid system popups blocking automation

The plugin also provides capabilities like iOS simulator screenshots and exporting the accessibility tree XML of the current foreground app.

Suitable Scenarios and Notes¶

Suitable scenarios:

Using full Xcode on macOS
DSH agent needs to automatically operate iOS simulators
Not wanting to rely on macOS accessibility authorization
Not wanting to rely on idb
Need to click any app, including system popups

Note the following limitations:

Requires full Xcode; testmanagerd is needed to launch XCUITest, and pure simctl launch cannot start the HTTP service
For real device WDA, you must resign the profile containing the device UDID and cannot reuse the simulator package directly
Simulator packages are unsigned, arch=arm64+x86_64, and can be reused across Macs
Android (ADB) support, element-level swipe / long press / text input are not completed in the roadmap
The plugin runs with the permissions of the current dsh process; you should check the source code and MIT license before installing
The community directory is an independent site, not equivalent to the official app store

Ending¶

dsh-mobile-control encapsulates the iOS simulator control capabilities driven by WDA into DSH tools. It is suitable for scenarios that require screenshots, finding elements, clicking, opening URLs, exporting accessibility trees, and privacy pre-authorization within an agent session.

dsh-mobile-control: DSH Plugin for driving iOS Simulator with WDA