ScreenClaw

Problem

Desktop UI automation is fragile because coordinates are rarely stable: window position, resolution, scaling, multiple monitors, and dialog layers can all break guessed clicks. ScreenClaw treats screenshots, window lists, and API responses as facts, then enforces a locate-act-verify loop instead of relying on intuition.

How It Works

• Window-level first: use get_window_list to find the target window, then locate buttons, inputs, and menus with the screenshot grid; use desktop_* APIs for desktop icons, system tray, and cross-window dragging.
• Falsify before clicking: for high-risk coordinates, zoom into the target with crop_zoom_screenshot, then verify with marker that the marked point actually lands on the element.
• Verify every action: re-screenshot after click, input_text, press_key, drag, etc.; when a new process or window appears, refresh get_window_list before verifying.
• Templates and self-checks: reusable flows can be saved as scenario templates with fixed windows and coordinates, while dynamic parts stay screenshot-based; on SELF_CHECK_REQUIRED, reload key context, summarize, and continue.

Boundaries

It depends on the local ScreenClaw service and readable screenshots. If the UI changes rapidly, coordinate labels are unclear, or the task needs real-time game-like input, first zoom, switch between window-level and desktop-level paths, or enter delegated mode. Templates are not absolute; when they fail, fall back to the general screenshot-locate-verify loop.