Preface

When performing image or screen-related agent tasks in DSH, a common problem is that while the model can “understand” the content, the provided positions are not precise enough. When subsequently performing annotation, measurement, frame differencing, or OCR, there is a lack of reproducible coordinate baselines. If an additional set of MCP servers are introduced, deployment and permission boundaries become more complex.

Below is an introduction to dsh-vision-primitives. It is a native visual reasoning plugin for DeepSeek Harness (DSH). It first converts images or screenshots into numbered grids, then parses the cells into precise coordinates, followed by local zooming and deterministic verification. When multimodal understanding is needed, it can connect to the MiMo V2.5 backend; when not needed, the local visual primitives can be used independently.

What is it

dsh-vision-primitives can be understood as a set of visual primitives in the DSH plugin ecosystem:

  • The core goal is precise pixel grounding: SOM grid, zoom, annotate, measure, diff, color find, OCR.
  • Provides MiMo V2.5 multimodal backend and registers mimo model routing, supporting streaming, function calling, and image input.
  • Does not depend on external MCP servers.
  • Repository owner is zouyuanqing, license is MIT.
  • GitHub address: https://github.com/zouyuanqing/dsh-vision-primitives

In one sentence: It breaks down “see image — select position — zoom — verify” into a set of callable, reproducible steps, rather than letting the model output coordinates based on intuition.

Core Capabilities

Pixel-level positioning

First frame, then grid, finally parse coordinates. The typical pipeline is:

vision_capture -> vision_grid -> read_image -> vision_resolve

Where:

  • vision_capture captures the screen or reads the workspace PNG to form a session frame.
  • vision_grid overlays a Set-of-Mark numbered grid, used to reduce error when the visual model outputs coordinates directly.
  • read_image is used to read image content.
  • vision_resolve parses the cells into precise pixel coordinates.

Local lossless zoom

vision_zoom performs local nearest-neighbor zooming and preserves the coordinate mapping chain back to the original frame. After this step, the model can continue to judge details within the zoomed image, while coordinates can still be mapped back to the original frame.

Deterministic verification

The plugin provides a set of tools for verification, suitable for pixel-level confirmation after “model judgment”:

vision_annotate / vision_measure / vision_ocr / vision_find_color
  • vision_annotate: Annotate.
  • vision_measure: Measure.
  • vision_ocr: OCR.
  • vision_find_color: Color find.
  • vision_diff: Frame difference, used to detect changed areas.

MiMo V2.5 Multimodal Backend

The plugin supports the MiMo V2.5 multimodal backend and registers the mimo model routing. This routing supports streaming, function calling, and image input.

Configuring an API key is required when using MiMo-related tools; pure local visual primitives do not require a key.

Chat Image Input

The plugin provides three chat image input modes:

  1. Native image attachments for visual models.
  2. Image bridging when sending via pure text models.
  3. paste-to-path text injection.

When sending, image bridging caches the image to .vispri/incoming-*.png and injects text similar to [Attached image: path], allowing text models to process it further based on the path.

Visual Evidence Protocol (VEP)

The plugin provides the Visual Evidence Protocol (VEP), outputting structured visual evidence, including:

  • caption;
  • layout;
  • elements;
  • normalized boxes and pixel boxes;
  • SOM cell IDs;
  • screen coordinates.

This layer of structured results is used to bridge fuzzy visual perception to subsequent deterministic coordinate operations.

Minimal OS Boundary

The plugin has a small OS boundary: it completes three types of operations only by calling Windows PowerShell system scripts via host subprocess:

  • screenshot;
  • OCR;
  • binary file persistence.

It does not contain desktop mouse/keyboard control.

Key Handling

The plugin does not hardcode keys. The MiMo API key is read lazily from DSH credentials.

Installation and Enabling

First, confirm the environment: The Node engine requirement is >=18. This package is a pure JS package with no build scripts, and does not require pnpm allowBuilds permission. Dependencies include:

  • @deepseek-ai/dsh-settings
  • @deepseek-ai/dsh-tools
  • @deepseek-ai/schemastery

The installation command is as follows:

dsh plugin --profile <name> add github:zouyuanqing/dsh-vision-primitives

Here, <name> is the profile placeholder; this article does not provide a specific profile name. After installation, start DSH to use the visual tools provided by the plugin in the current session.

If a specific version installation is needed:

dsh plugin --profile <name> add github:zouyuanqing/dsh-vision-primitives#<commit-sha>

If installed via the npm package:

dsh plugin add dsh-vision-primitives

If using the MiMo backend, configure the API key:

dsh credentials set MIMO_API_KEY <your-key>

The configuration parsing priority is:

WebUI user settings > row config > defaults

Within this, the lookup order for apiKey is: check settings first, then credentials.

Note that the installation type affects the availability of the WebUI configuration card: The WebUI config card is only available under the bundle install form. For the dynamic plugin sandbox type, because the approval policy is never, the Client half cannot be activated; in this type, credentials can be configured via CLI.

Typical Usage

An interactive visual reasoning workflow can be written as:

vision_capture -> vision_grid -> read_image -> vision_resolve -> vision_zoom -> vision_annotate / vision_measure / vision_ocr / vision_find_color -> vision_diff

Understand step by step:

  1. vision_capture first converts the screen or workspace PNG into a session frame.
  2. vision_grid overlays the numbered grid.
  3. read_image reads the image content.
  4. vision_resolve parses the target cell into precise coordinates.
  5. vision_zoom zooms in on a local area, preserving the original frame coordinate mapping.
  6. Use vision_annotate, vision_measure, vision_ocr, vision_find_color for verification.
  7. When detecting changes is needed, use vision_diff.

Frame files are stored in:

sandboxPolicy.workspaceRoot/.vispri

The incoming file path generated by image bridging during sending looks like:

.vispri/incoming-*.png

If you want to run kernel tests:

node kernel-test.js
node inflate-diff-test.js

Applicable Scenarios and Notes

Suitable for:

  • Processing screenshots or images in DSH agents.
  • Needing to ground visual results to precise pixel coordinates.
  • Needing deterministic verification such as annotation, measurement, frame differencing, color finding, or OCR.
  • Wanting to reduce dependency on external MCP servers.
  • Optionally connecting to MiMo V2.5 for multimodal understanding.

Note:

  • The plugin runs with the current dsh process permissions; source code and license should be checked before installation.
  • Screen capture and native OCR are currently Windows implementations, using PowerShell Graphics.CopyFromScreen and WinRT OcrEngine.
  • The plugin does not include desktop mouse/keyboard control.
  • MiMo tools require an API key; pure local visual primitives do not require a key.

Conclusion

The value of dsh-vision-primitives lies in transforming visual reasoning from “model directly guessing coordinates” to an engineering process of “grid — parse — zoom — verify”, and optionally connecting to the MiMo V2.5 multimodal backend.

GitHub: https://github.com/zouyuanqing/dsh-vision-primitives. This article does not use unverified community directory page URLs; if a directory page is needed, please refer to the actual entry provided by the current plugin directory.