Introduction

When developing agents, running code on real Android devices is a common requirement. Phone-Eye aims to solve this problem. It provides visual and interaction capabilities on devices through adb, enabling AI agents to “see” the screen and perform actions such as taps, swipes, and other operations. This is achieved by fusing a vision model with the UI tree (Accessibility Tree), and it works with any MCP client.

What This Is

Phone-Eye is an open-source MCP plugin that enables AI agents to directly see and operate real Android phones via adb. It does not require installing additional apps on the phone, root access, or an extra server.

  • Core capability: vision + UI tree fusion.
  • Supported clients: any MCP client (Claude Code, Codex, DSH, etc.).
  • Maintainer: boheastill.
  • License: MIT.

Core Features

Phone-Eye provides the following core capabilities:

  1. Visual perception: capable of capturing screenshots and reading on-screen content.
  2. Screen interaction: supports taps, swipes, and typing.
  3. Stability: supports automatic reconnection after Wi-Fi ADB disconnects, ensuring a stable connection.
  4. Unattended operation: supports 24/7 unattended operation; when unexpected pop-ups appear, the agent reads and handles them.
  5. Multi-device support: supports connecting multiple phones; each phone requires running a separate phone-eye process, distinguished by setting the ANDROID_SERIAL environment variable.

Installation and Setup

Before installation, make sure Python 3.10+ and git are installed on your computer.

  1. Clone the repository and install dependencies:
    git clone https://github.com/boheastill/phone-eye && cd phone-eye && pip install -r requirements.txt
  1. Configure environment variables (e.g., set the vision model API key):
    export PHONE_EYE_VISION_API_KEY=<your_api_key>
  1. Start the service:
    python server.py

Typical Use Cases

Phone-Eye is suitable for the following scenarios:

  • Mobile web QA loops: verify an agent’s results on a real screen.
  • OEM setup wizards: handle various dialogs that appear when a device starts up.
  • Form regression checks: automatically test form filling and validation.
  • Unattended sentinel: let an agent monitor overnight, wake the device, unlock it, interact with it, and capture screenshots.

Applicable Scenarios and Caveats

When deploying Phone-Eye, note the following limitations and considerations:

  • Lock screen limitation: When the screen is locked, it can be read but cannot be operated.
  • IME limitation: Typing supports ASCII characters only; Chinese input requires using clipboard techniques.
  • New device control: A brand-new phone cannot be fully controlled until USB debugging is manually enabled.
  • Initial authorization: When connecting a phone to a new computer, the first “Allow USB debugging?” dialog must be tapped manually once.
  • Vendor ROM limitations: Some vendor ROMs (such as MIUI) restrict input operations while the screen is locked.
  • Operating system: iOS is currently not supported.
  • Visual quality: Visual results depend on the model used.
  • Deployment requirements: No app installation is required on the phone, no root access is needed, and no extra server is required.

Conclusion

Phone-Eye combines a vision model with the UI tree, providing AI agents with the ability to operate real Android devices. It is a practical tool for agent development scenarios that involve automated testing, device operation, or physical interaction. For more details and usage examples, refer to its GitHub repository.