About this skill

Problem

macOS desktop automation can break when permissions, UI changes, or app state are not stable. macOS Desktop Controller targets macOS 14.x and uses Python with AppleScript, screenshots, and OpenCV template matching to turn “open an app, click an element, type text, and run a flow” into composable modules.

Core capabilities

App control: launch, quit, or restart apps; move, resize, minimize, or maximize windows.
Precise interaction: click by coordinates, element ID, or vision; keyboard input, hotkeys, drag, and scrolling.
Vision control: capture full-screen, window, or region screenshots; identify elements and match templates.
Workflow management: chain tasks with condition branches, loops, error handling, and retries.

Boundaries

It fits scripted UI automation on local macOS, especially when window state, image matching, or repeated flows are needed. Because it depends on system permissions and desktop UI, element targeting can be affected by occlusion, resolution, DPI, or dynamic interfaces. Use debug mode and explicit retry policies for passwords, sensitive data, or irreversible actions.

Screenshots

Use Cases

Launch, quit, or restart target macOS apps locally, then adjust window position, size, and minimized state.
Click buttons by coordinates, element ID, or vision, then type text, send hotkeys, drag files, or scroll pages.
Capture full-screen, window, or region screenshots and use OpenCV template matching to confirm buttons or states.
Chain app control, interaction, and vision checks into workflows with condition branches, loops, retries, and error handling.

Best For

Engineering teams scripting macOS app windows, clicks, and input with Python.
Automation engineers turning local UI steps into repeatable flows with retries and conditions.
QA or test personnel verifying desktop element states through screenshots and template matching.
Platform engineers debugging AppleScript, OpenCV, and Python desktop UI integration chains.