dsh-android-pane

Watching an AI agent tap through an Android app is usually a black box: you hear it narrate its actions but cannot see the screen or jump in when it goes off track. dsh-android-pane drops a live Android screen panel right beside your DSH chat. Emulators and USB-connected real devices both work, with zero configuration and no DSH config changes needed; just open the web profile and you are watching.

The video path pushes scrcpy-server onto the device, encodes H.264, forwards it over adb port forwarding to the local loopback, and renders it with WebCodecs hardware decode at 30 to 60 fps. If scrcpy cannot start on a legacy device or hardened system, it degrades gracefully to a screenshot carousel. The agent side ships context-efficient tools for tap, swipe, text input, key press, scroll, and rotate, each auto-returning a screenshot so the agent can verify its own action. A uiautomator control tree with clickable coordinates lets the agent locate elements by fuzzy text match instead of blind coordinates. When it hits a FLAG_SECURE page such as a banking or password-manager app, the panel detects the blackout, runs a mitigation chain, and falls back to a tappable control-tree text view with every step made explicit. You can also tap and drag on the live panel yourself while the agent is working; a red badge lights up so everyone knows who is driving.

Ideal for developers and QA engineers who need an agent to operate Android apps end to end, and for everyday DSH users curious about watching an AI actually see and act. All media streams are bound to 127.0.0.1 only, keeping the privacy boundary clean.