Windows Desktop Control

Problem

Many Windows apps, such as WeChat, QQ, and Electron apps, render UI through CEF/Chromium instead of standard Win32 child controls. As a result, pyautogui.click() can miss targets because of focus and hit-test differences. This skill targets the desktop loop of capturing a screenshot, analyzing it with a vision model, performing mouse and keyboard actions, then verifying the result.

How It Works And Limits

• Screenshot: captures a window or full-desktop image.
• Vision parsing: asks GLM-4V-Flash for percentage positions instead of exact pixel coordinates.
• Control: uses win32api.SetCursorPos, mouse_event, and keybd_event for low-level clicks and keys.
• Text input: short ASCII can use pyautogui.write(); Chinese or emoji should be pasted via clipboard to reduce IME interference.
• Verification: takes another screenshot and asks the vision model to confirm the action.

It is not intended for terminal, file I/O, or web scraping. Window handles may become invalid, Enter may send or insert a newline depending on app settings, and focus can be stolen, so delays, window re-enumeration, and final screenshot checks are important.