AI-Eyes Screen Vision Plugin 2.0

Problem

Text-only AI often understands visible text but struggles with graphical buttons, icons, status bars, floating tooltips, or dense controls. Screen actions need element location to become checkable and reproducible rather than estimated.

How it works

• Vision pyramid: starts from L0 full-screen scanning and zooms into L3 micro regions, pushing location precision to 0.125px.
• Candidate filtering: supports exact, fuzzy, semantic, and position modes to narrow candidates by exact text, approximate characters, semantic similarity, or weighted screen position.
• Coordinate correction: the coordinate transformer compensates for zoom scale, so returned coordinates map to the real screen instead of a cropped or intermediate image.
• Optional validation: with Ollama and LLaVA, candidate elements can be rechecked by a vision-language model, reducing OCR misreads and visually similar characters.

Boundaries

It depends on Pillow, rapidocr-onnxruntime or easyocr, and a screenshot source such as pyautogui. For icon-heavy, sparse-text, blurry, or very high-resolution interfaces, prefer L2/L3 levels and enable semantic validation.