Introduction

In DSH agent scenarios, text models often need to process images: users send screenshots, web error screenshots, tables, long chat logs, or want the model to find a specific icon, button, or text area within the image. If the underlying model does not support images natively, the plugin can only rely on external patches, manual descriptions, or converting images to text before passing them to the model.

dsh-vision-skill is a DeepSeek Harness (DSH) standard plugin for image recognition skills. It packages image recognition capabilities into a native DSH runtime skill, allowing pure text models that do not support images to view, OCR, and locate targets in images.

What is this

dsh-vision-skill is maintained by DDDFXYqiming and is licensed under MIT.

It provides the DSH native runtime skill vision, which is loaded on demand by the model and progressively exposes image recognition tools. Once loaded, image-related requests in the conversation can be routed to corresponding local tools rather than requiring the text model to directly understand the image.

This plugin is suitable for plugin-based agent workflows within the DSH ecosystem, especially for scenarios that require supplementing image recognition, text extraction, and target localization capabilities to text models.

Core Capabilities

Below are the verified tool capabilities.

vision_analyze

vision_analyze is used to recognize local images and supports 6 modes:

  • general
  • ocr
  • table
  • code
  • error
  • evidence

Among them, evidence is a structured evidence mode that returns a JSON snippet containing:

summary / ocr_full_text / layout / semantics / uncertainty / visual

This mode is suitable for scenarios where image evidence needs to be split into referenceable fields, rather than just obtaining a natural language description.

vision_ocr

vision_ocr is a standalone OCR tool that preserves original layout. It is suitable for extracting text from images while maintaining the original structure as much as possible.

vision_ground

vision_ground is used for target localization. The model can request to find a specific target in the image, and the tool returns pixel coordinates and normalized coordinates.

vision_detect

vision_detect is used to enumerate a class of elements, such as buttons, icons, table cells, etc., and provide numbering and pixel coordinate boxes.

vision_dominant_colors

vision_dominant_colors is used for dominant color analysis. This capability uses a local pixel algorithm and does not consume the visual API.

vision_long_screenshot_ocr

vision_long_screenshot_ocr is used for chunked OCR of ultra-long screenshots. The process runs local tesseract first, falls back to VLM (Vision Language Model) if it fails, and finally merges the results.

vision_clipboard

vision_clipboard is used for fallback recognition of clipboard images. It is suitable for scenarios where the model is asked to “view the image” directly after taking a screenshot.

Engineering Capabilities

In addition to the tools themselves, dsh-vision-skill also includes some engineering designs.

Direct Image Paste

Starting from v0.4, direct image pasting uses paste-to-path and no longer requires the pi-ai patch.

If the old host does not have the reference capability, direct pasting will automatically fall back to inserting path text.

Multi-provider Failover

The plugin supports multi-provider failover. When encountering a 429, it will retry once with Retry-After backoff.

Image Caching

The plugin provides image memory caching, caching based on the image SHA-256 plus mode / budget / crop / prompt. When hit within TTL, it returns directly:

{
  "cached": true
}

Path Fencing

The image path must be located in the session workspace, the DSH attachments directory, or one of the allowedDirs, and must pass realpath validation.

This prevents arbitrary path images from being read directly.

Installation and Usage

The installation command is as follows:

dsh plugin --profile web add github:DDDFXYqiming/dsh-vision-skill

This command installs the plugin into the web profile.

Runtime environment requirements:

{
  "engines": {
    "node": ">=22.19"
  }
}

For keys, it is recommended to use DSH Credential references:

credential: VISION_API_KEY

The plugin is also compatible with plaintext apiKey, but it is not recommended.

When customizing configuration, use bare entries to override by ID. If duplicate IDs appear, it triggers:

duplicate loader entry id

The host will crash directly upon startup, so it is not recommended to insert another entry with the same ID in a configuration that already has a vision-skill entry.

Typical Usage

Simply describe the requirement in natural language within the conversation; the request will be routed to the corresponding tool.

识别这张图 <路径>

Corresponding tool:

vision_analyze

OCR 这张图 <路径>

Corresponding tool:

vision_ocr

在这张图里找到 <目标>

Corresponding tool:

vision_ground

Returns pixel coordinate boxes.

清点这张图的所有按钮

Corresponding tool:

vision_detect

这张图的主色是什么

Corresponding tool:

vision_dominant_colors

This tool uses a local algorithm and does not consume the visual API.

提取这段长聊天记录的文字

Corresponding tool:

vision_long_screenshot_ocr

看图（剪贴板截图）

Corresponding tool:

vision_clipboard

Applicable Scenarios and Notes

dsh-vision-skill is suitable for the following scenarios:

  • Text models need to recognize image content.
  • Need to extract text from images.
  • Need to locate targets or enumerate elements in images.
  • Need to process ultra-long screenshots.
  • Need to analyze dominant colors of images.
  • Want to manage image recognition capabilities in DSH using native plugins.

Things to note before use:

  • The plugin runs with the current dsh process permissions. It is recommended to check the source code, dependencies, and license before installation.
  • The plugin registers the skill name vision at the runtime layer. If a skill with the same name is installed at the user or project layer simultaneously, they may shadow each other; it is recommended to install only one.
  • Direct pasting belongs to client/Web behavior. After restarting the web host, you need to hard refresh the browser to verify.
  • The pi-ai old patch is only kept for compatibility. If the old patch is reapplied after a dsh upgrade, you can run:
scripts/restore_pi_ai_vision_patch.py
  • Image paths are restricted by the workspace, DSH attachments directory, or allowedDirs.
  • When customizing configuration, avoid duplicate IDs, otherwise the host will fail to start.

Links

GitHub:

https://github.com/DDDFXYqiming/dsh-vision-skill

License:

MIT