PC-Control Computer Control Assistant

Problem

Local desktop automation often gets stuck when a model can see the screen but cannot operate it reliably: screenshots, coordinates, windows, app launching, file access, and CMD commands are scattered across separate scripts without a clear action boundary. PC-Control consolidates common local control actions into one AI Agent skill, allowing the model to interpret natural language into the right local operation and emit auditable status before and after execution.

Core Capabilities and Workflow

The skill is built on Python 3.11, pyautogui, and pywin32, covering:
- Screen awareness: screenshot, analyze_screen, cursor_pos, screen_size
- Mouse and keyboard control: mouse_click, mouse_move, mouse_drag, key_input, key_press, key_combo
- Windows and processes: window_list, window_focus, window minimize/maximize/close, process_list
- Files and commands: file_list, file_read, file_write, file_delete, cmd_exec

The workflow first parses the instruction, then classifies the action as self-executable, allowed inside a whitelist, requiring confirmation, or blocked by a blacklist. App launch/kill, file deletion, and CMD commands must output a 📋 or ⚠️ report first, execute only after confirmation, and finish with an ✅ result log.

Boundaries

It fits controlled automation in a Windows local environment: launching whitelisted apps, operating whitelisted directories, and running filtered commands. It should not touch privacy directories, upload private files, or execute untrusted commands without explicit confirmation. The practical focus is to treat “what may be done” and “what must be reported” as engineering constraints, not as an open-ended local executor.