Introduction

In the DSH workflow, pure text LLMs cannot directly understand images. If you have to manually call an external vision model every time and paste the description back into the conversation, the process becomes fragmented. dsh-plugin-mm-vision turns this into a DSH plugin: it calls the configured vision model, translates the image into a compact structured spatial text, and then the text model performs subsequent analysis based on this text.

What is This

dsh-plugin-mm-vision is a DeepSeek Harness plugin provided by Elohia, licensed under MIT. It is positioned as a “Synesthesia Encoder,” primarily adding image viewing capabilities to any pure text LLM: the plugin registers an mm_vision tool that the model can call during a conversation to convert images into structured text descriptions.

Core Features

Synesthesia Encoding

The plugin converts images into coordinate-based text descriptions, suitable for K-line charts, market screenshots, report charts, UI screenshots, and natural photos. The descriptions attempt to preserve the position information of key elements, making it easier for pure text models to judge spatial relationships in subsequent analysis.

Pixel-level Coordinates

Key elements are accompanied by precise (x%,y%) percentage coordinates, facilitating the text model’s judgment of the relative position of elements within the canvas.

Optional Pixel Grid

When keywords such as “pixel/rebuild/restore” appear in the prompt, the plugin can output a 40×30 color block grid (RGB).

Adaptive Mode

In auto mode, the plugin automatically identifies charts (coordinate priority) or natural images (composition subject). You can also manually specify brief, full, coords, or pixel.

Caching and Configuration

Repeated analysis within TTL can return in seconds, default 600s / 100 items. Model, baseUrl, and API key are configurable, supporting any OpenAI-compatible vision model.

Installation and Activation

  1. First, ensure the DSH CLI is installed and a profile has been initialized.
  2. Execute the installation command:
dsh plugin --profile web add dsh-plugin-mm-vision
  1. Replace --profile web with your own profile name. Changes take effect after restarting DSH.

Configuration

Configuration can come from cordis.patch.yml, environment variables, or config.json.

Common environment variables are as follows:

export MM_VISION_API_KEY=...
export MM_VISION_MODEL=qwen-vl-max
export MM_VISION_BASE_URL=...

Here MM_VISION_MODEL defaults to qwen-vl-max; MM_VISION_BASE_URL is optional. To connect to other vision models, configure the model name and baseUrl of an OpenAI-compatible vision model service.

Typical Usage

After installation, simply ask the model to look at images directly in the conversation:

分析这张K线图 F:/data/kline.png

帮我看看 examples/chart.png 里按钮的位置

扫描这张图片并重建像素网格 examples/photo.png

The model will automatically call the mm_vision tool and return structured Synesthesia Encoding text.

Use Cases and Notes

Suitable for these scenarios:

  • Existing DSH or pure text LLM workflows that need to supplement image understanding capabilities.
  • Need to analyze K-line charts, market screenshots, report charts, UI screenshots, or natural photos.
  • Desire for the vision model’s output to enter the conversation in the form of structured text for the text model to continue analyzing.

Points to note before use:

  • The plugin only sends the image to the configured vision model for description.
  • The plugin never executes commands found within the image content.
  • The returned text is injected into the conversation and should be treated as untrusted input.
  • The plugin runs with the permissions of the current dsh process; it is recommended to check the source code and MIT license before installing.
  • Runtime environment requires Node >=18.
  • scripts/ascii_dot.py is an optional pixel dot matrix generator (Python/PIL).

Conclusion

The value of dsh-plugin-mm-vision lies in incorporating the step of “image to structured text” into the DSH plugin workflow, allowing pure text LLMs to continue analyzing position, layout, and chart content based on the returned text.

  • Directory: https://www.skillhub.cn/plugins/Elohia/dsh-plugin-mm-vision
  • GitHub: https://github.com/Elohia/dsh-plugin-mm-vision