AutoPilot Composer

Problem

AutoPilot Composer targets long mixed workflows that span desktop applications and browser pages: opening a local program, copying or exporting data, filling web forms, capturing screenshots, and repeating the sequence in batches. Hand-writing task_flow.json is tedious, while ad-hoc scripts often fail halfway and then require a full rerun, brittle locators, and separate handling for GUI and web actions.

How it works

The skill replays actions as structured steps through two engines: cdp_engine.py uses Chrome CDP for open_url, input_text, click_elem, screenshot, hover, drag, key_press, and upload_file; gui_engine.py uses pyautogui for launching applications, coordinate clicks, drag-and-drop, hotkeys, and image-based clicking.

Recording is provided in three paths: the web recorder captures clicks, inputs, drags, hovers, keyboard events, file uploads, and navigation, then exports task_flow.json and recorded_flow.js; the desktop recorder targets Windows applications and outputs desktop_flow.json plus a standalone pyautogui script; the merged session recorder interleaves web and desktop actions chronologically and can export a unified flow along with SOP.md. During replay, main_task.py retries failed steps according to config.json and stores progress in breakpoint.json, so execution can resume after a fix. v3.2+ adds elements.json, preset_elements.json, and operation_log.json, using multi-strategy locators and structured logs to reduce maintenance cost when pages change.

Boundaries

The skill genuinely controls the mouse, keyboard, and browser, so it should run in controlled environments or test accounts. Desktop coordinates and click_icon are sensitive to resolution and window placement; CDP usually attaches to an existing ordinary tab, and open_url does not open a new tab; upload_file records only the file name, while the replay side must provide the disk path. Generated :nth-child selectors should be reviewed and replaced with more stable #id or semantic selectors where possible.