Introduction

The plugin architecture of DeepSeek Harness (DSH) allows native functionality to be extended. In the model inference chain, if the currently routed model does not support image input (such as some text-only models), directly sending image blocks will trigger a rejection error: model does not support image input.

The dsh-vision-assist plugin intercepts the LLM call boundary and converts images into text instructions, enabling text-only models to “see” images. In essence, it gives a main model without visual capabilities a switchable multimodal “sidecar.”

Plugin Overview

  • Plugin name: dsh-vision-assist
  • Maintainer: wjling
  • Category: Model inference
  • License: MIT

Core Features

1. vision_recognize Tool

The plugin registers the vision_recognize tool at the host level. Any session can directly call this tool to recognize local images using the selected multimodal model.

2. Image Fallback at the Model Call Boundary

When the session’s routed model does not support image input, the plugin intercepts the request at the ctx.llm.prepareCall / ctx.llm.stream boundary:

  • Clones the request object.
  • Replaces image blocks (including tool-role messages) with the text of “attachment store path + vision_recognize guidance.”
  • The main model accepts the request normally and calls the vision_recognize tool.
  • ctx.llm.stream connects directly to the selected multimodal model to obtain the recognition text, then the task continues.

3. Hot Configuration Updates and UI

The plugin provides a settings page UI that supports switching the recognition model and timeout in DSH Settings → Plugins → “Vision Assist vision-assist” card. Configuration changes take effect immediately after saving, without restarting DSH.

Installation

Ensure your DSH version is >= 0.1.7-alpha.1. Run the following command to install:

dsh plugin --profile web add github:wjling/dsh-vision-assist

After installation, restart DSH to activate the plugin.

Configuration

The plugin supports two configuration methods:

Method 1: Settings Page UI (Recommended)

  1. Open DSH Settings.
  2. Go to the “Plugins” panel.
  3. Find the “Vision Assist vision-assist” card in the “Official” group.
  4. On the detail page, enable the master switch, then select the recognition model (it must be configured in llm-pi-ai) and the timeout.
  5. Click save. The configuration takes effect immediately.

Method 2: Profile Patch

Add the following to ~/.dsh/profiles/<profile>/cordis.patch.yml:

- id: vision-assist
  config:
    enabled: true
    provider: <provider_name>
    model: <model_id>
    timeoutMs: 300000

Note: The selected model must declare input: [text, image] in llm-pi-ai.providers.<provider>.models; otherwise, DSH will reject image input for that model.

How It Works

The process is as follows:

  1. The user sends an image in the input box.
  2. api-session-controller performs a modality pre-check. If the model supports images, the request enters the session normally, and the GUI displays the original image.
  3. When the main model initiates an LLM call, the plugin checks whether the model supports visual input.
  4. If it does not, the plugin clones the request and rewrites the image blocks into text instructions (including the local attachment path).
  5. The main model receives the instructions and calls the vision_recognize tool.
  6. vision_recognize triggers the vision model to recognize the image and return text, after which the main model continues processing the task.

Notes

  • Original Image Storage: All uploaded original images are always stored in ~/.dsh/attachments/v1/objects/, using content addressing.
  • UI History Retention: GUI bubbles and session history always retain the original images. Only the request sent to a non-visual model is rewritten into recognition guidance.
  • Compatibility: Requires Node >= 20 and DSH >= 0.1.7-alpha.1.

Summary

dsh-vision-assist solves the problem of text-only models in DSH being unable to handle image input. Through tool calls and request rewriting, it adds visual capabilities to the model without modifying the existing request pipeline. It is suitable for scenarios where multiple models are switched between while maintaining unified visual processing.