Windows Desktop Automation

The Problem

Some Windows tasks live at the GUI layer: they require coordinate-based clicks, drags, scrolling, typing Chinese into a focused window, enumerating or switching windows, or capturing screen content and sending it to teammates. These workflows usually cannot be completed through file APIs or web interfaces alone. windows-auto provides a set of desktop-oriented scripts that split mouse, keyboard, window management, screenshotting, and file sending into composable steps.

How It Works

Core capabilities include:

• Mouse control: scripts/mouse.py supports moving, clicking, double-clicking, dragging, scrolling, and taking screenshots, which is useful for repetitive GUI operations.
• Keyboard input: scripts/keyboard.py supports text entry, single keys, and hotkey combinations; Chinese input is handled through the clipboard, so the target window must already be focused.
• Window management: scripts/window.py can list, find, focus, maximize/minimize, close, and move windows, making it easier to orchestrate multi-app workflows.
• Screenshots and sending: scripts/screenshot_powershell.py uses PowerShell to call .NET's System.Windows.Forms and System.Drawing for zero-dependency screenshots, including full-screen, region, and multi-monitor capture; scripts/screenshot_and_send.py captures the full screen and prepares it for group chat; scripts/send_file.py can send Word, PDF, images, and other files.

In practice, it is typically used by running scripts inside an automation workflow: confirm the window state and coordinates first, then invoke mouse, keyboard, or window scripts, and finally output results through screenshot or file-sending scripts. The reference notes also cover coordinate systems, common hotkeys, typical flows, and emergency stop methods.

Boundaries and Caveats

• Python screenshots and mouse workflows depend on pyautogui and pillow; PowerShell screenshots are suitable for environments where extra Python dependencies should be avoided.
• Some actions, such as UAC prompts, may require administrator privileges.
• Chinese input relies on the clipboard, so unfocused target windows can receive input in the wrong place.
• If an operation goes wrong, move the mouse quickly to the top-left corner (0,0) to abort automation.