Rebind PC Auto Control

Problem It Solves

When a task depends on a GUI, legacy software, OS settings panels, or browser front-end workflows, plain CLI or APIs may not be enough. Manual clicking is repetitive, while hand-written scripts struggle when the interface changes. This skill targets cases where an Agent needs to see the screen and operate the mouse/keyboard, turning local GUI automation into callable MCP tools.

How It Works

It uses Rebind, a programmable input layer, together with the official MCP Server to expose a “perceive-plan-act-verify” loop. A typical cycle is:
- Perceive: call screenshot, screenshot_window, or zoom to capture the current screen state.
- Plan: infer the target element, text to type, scroll position, or window switch from the image and goal.
- Act: use move_mouse, click, type, key, scroll, window, or run_lua for more complex orchestration.
- Verify: take another screenshot to confirm the effect, then re-plan if the result is not expected.

Compared with solutions that require paid accounts or specific OS sessions, it focuses on a local loopback WebSocket and cross-platform operation on Windows, macOS, or Linux within a trusted desktop.

Boundaries and Notes

The default setup is for local, controlled environments: remote access should listen only on the loopback address, and screenshots are used for local decisions. For actions involving payment, deletion, outbound sending, or password entry, human-in-the-loop confirmation is recommended. Destructive operations are blocked by default; complex flows should still review run_lua and tool parameters before delegating irreversible steps to full automation.