dsh_omnivision

Driving a real desktop GUI from a language model has always hit a wall: you either need an expensive multimodal vision model, or the model simply cannot see the screen. OmniVision sidesteps that with OmniParser, which converts a screenshot or any image into structured elements — text, icons, and interactive controls — each tagged with pixel coordinates. The result is a plain-text representation of the screen that any DeepSeek Harness conversation model can reason over, no extra visual capability required.

On top of that, the plugin registers a full set of gui_* tools: gui_capture takes a screenshot and parses it, gui_act performs real mouse and keyboard actions (click, drag, type, hotkey, scroll) by element ID or raw coordinates, gui_find searches the last capture, gui_verify re-captures until a target text appears or disappears, and gui_task chains multi-step UI plans with re-parsing between steps. The browser-side OmniVision dock gives you a live SOM-annotated view with hover highlight and click-to-zoom, a recognition history with added/removed/moved diffs, a one-click summary, and local image parsing — so you can always see what the model perceived and what it did, without digging through logs.

It is built for developers and power users who want to orchestrate desktop automation through conversational AI in DeepSeek Harness, especially those who want to keep inference costs and latency down by avoiding dedicated multimodal endpoints.