Introduction

The extension mechanism of DSH is “everything is a plugin”. When a model needs to operate on a real interface, Android and Web often require separate handling of the target, location, and runtime environment. ciky20171114/dsh-plugin-midscene provides DeepSeek Harness (DSH) with a Midscene-based AI-driven UI automation: the model sees the screen, locates elements based on natural language descriptions, and acts on real Android devices or real Chrome pages.

It provides two tools, android_ui and web_ui, via a ctx.midscene capability seam.

What is this

dsh-plugin-midscene is a DSH plugin maintained by ciky20171114 under the MIT license.

Its capability structure can be summarized as:

  • Single seam: ctx.midscene
  • Two providers: Android and Web
  • Two tools: android_ui and web_ui

Core Features

  • The Android provider targets an ADB-connected device.
  • The Web provider targets an active page of a running Chrome; the provider only connects and does not launch Chrome.
  • android_ui and web_ui use a single action parameter and support tap, act, input, query, assert, boolean, and back.
  • The vision model is configured via environment variables to be a Midscene-compatible model; documentation indicates this is not configured via DSH’s ctx.llm.

Installation and Activation

First, prepare the environment:

  1. DSH CLI and a profile.
  2. Android scenario: adb devices can see the device.
  3. Web scenario: Chrome is started with --remote-debugging-port=9222 --user-data-dir=<dir>; the plugin only connects and does not launch Chrome.

Installation:

dsh plugin --profile mysetup add dsh-plugin-midscene

After installation, add exactly one provider row in the profile’s cordis.patch.yml. Example path:

~/.dsh/profiles/mysetup/cordis.patch.yml

In the same context, two providers cannot both possess ctx.midscene, so only one provider row can be configured.

Android Example:

- insert:
    - id: midscene-android
      name: dsh-plugin-midscene/android
      config:
        deviceId: ''
        aiActionContext: ''

An empty deviceId means taking the first device from getConnectedDevices(); aiActionContext is a free context.

Web Example:

- insert:
    - id: midscene-web
      name: dsh-plugin-midscene/web
      config:
        browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser/<id>'
        aiActionContext: ''

browserWSEndpoint can be obtained from the webSocketDebuggerUrl of http://127.0.0.1:9222/json/version. After Chrome restarts, the ID changes, and the provider row needs to be updated and dsh restarted.

Launch:

dsh --profile mysetup

If port 3080 is occupied, you can add:

dsh --profile mysetup --port 3081

For local development, you can use:

dsh plugin --profile dev add /path/to/dsh-plugin-midscene

Model Configuration

Configure the Midscene-compatible vision model via environment variables:

export MIDSCENE_MODEL_NAME=glm-4.6v
export MIDSCENE_MODEL_BASE_URL=https://open.bigmodel.cn/api/paas/v4/
export MIDSCENE_MODEL_API_KEY=<your key>
export MIDSCENE_MODEL_FAMILY=glm-v

Tools and Error Semantics

android_ui and web_ui share a single action parameter, which is handled by the tool after the model selects the action. Verified actions include:

tap
act
input
query
assert
boolean
back

A failed assertion is treated as a successful result, manifested as pass: false; only infrastructure failures trigger the error path.

Design Boundaries and Limitations

The design boundaries of this plugin are “no policy, no recovery”:

  • No retry
  • No precondition checks
  • No automatic recovery

Other limitations:

  1. Each provider instance supports only one target.
  2. No reconnection; disconnection mid-session is manifested as a rejected call.
  3. @midscene/android and @midscene/web are fixed at exactly 1.11.0.
  4. The Web puppeteer is a peer dependency; Chrome is provided by the deployer and is not downloaded by the plugin.
  5. The Web provider disconnects during teardown without closing, and the Chrome process is kept by the deployer.

Troubleshooting Installation

pnpm ≥ 10 might report ERR_PNPM_IGNORED_BUILDS due to transitive install scripts from sharp / @ffmpeg-installer/linux-x64. A allowBuilds fix is required per profile: open:

~/.dsh/profiles/<name>/pnpm-workspace.yaml

Adjust relevant items according to the allowBuilds hints from pnpm output, then re-run the add command.

Use Cases and Considerations

Suitable for scenarios where DSH is already in use and the model needs to operate on real Android devices or Chrome pages.

Recommendations before use:

  • The plugin runs with the permissions of the current dsh process; check the source code and license before installation.
  • It is not responsible for launching Chrome and does not automatically recover abnormal UI states.
  • The Web scenario requires the deployer to provide Chrome and resolve puppeteer in the environment.
  • The vision model needs to point to an accessible service via environment variables.

Links

  • Plugin Directory: https://www.skillhub.cn/plugins/ciky20171114/dsh-plugin-midscene
  • GitHub Repository: https://github.com/ciky20171114/dsh-plugin-midscene