Preface¶

DeepSeek Harness (DSH) determines whether to allow GUI image attachments based on the inputModalities declared by the model. Chat-completions lines like DeepSeek-V4-Pro and standard Flash are pure text models; pasting an image directly when they are selected will be natively rejected. Community vision plugins offer tools like view_image, but those are designed for file paths or URLs; image blocks attached mid-conversation still won’t work with pure text models.

dsh-vision-proxy, maintained by Flyvhidbwo, registers a deepseek-vision provider route in DSH. It declares support for image input externally, translates each attached image into text via a VLM within the request stream, and then delegates to the actual DeepSeek adapter for the response. The conversation brain remains DeepSeek; image recognition is an added bridging capability. The plugin is currently at version 0.4.1, MIT licensed, has 14 GitHub stars, and is categorized under model inference.

It’s important to note that since dsh 0.1.1, official vision models (e.g., DeepSeek-V4-Flash-Vision-Exp) are natively multimodal—you can send images directly without this plugin. This plugin primarily targets scenarios involving image recognition with Pro/text Flash, local Ollama, and custom OpenAI-compatible VLMs. The plugin has been adapted for the adapter prepareCall interface in dsh 0.1.1-rc.2.

What Is This¶

dsh-vision-proxy is a bundle plugin for DeepSeek Harness. It wraps existing DeepSeek adapters, registers a deepseek-vision route (displayed as DeepSeek + Auto Vision in the model selector), allows image attachments during the pre-check stage, and then uses a Vision-Language Model (VLM) to translate image content (OCR, layout, details) into plain text prefixed with `[Image Transcription], which is then passed to DeepSeek to generate a response.

The data flow is as follows:

User attaches image ──▶ deepseek-vision route ──▶ Translation via VLM (OCR + Layout + Details)
                    │                        │
                    ▖                        ▖
            DeepSeek responds ◀── Plain text conversation (image replaced by [Image Transcription] text)

GUI image attachments are transcribed using the official deepseek-v4-flash-vision-exp by default (costs about 1/3 of V4-Pro); you can also configure it to use any OpenAI-compatible VLM such as Bailian, Zhipu, OpenRouter, etc., or rely on autoLocalOllama to automatically detect a local Ollama instance.

Core Features¶

Routing and Translation¶

• Registers a providerId: deepseek-vision, internally delegating to innerProvider: deepseek-official.
• Each GUI image block is translated via a VLM before entering the DeepSeek conversation; the native read_image tool is also available under the same route.
• Translation results are cached in-process based on the image byte SHA-256 hash (up to 200 entries). The same image is translated at most once per process.

Multi-Backend and Fallback¶

• Supports any OpenAI-compatible /chat/completions endpoint: Bailian/Qwen, QwenCloud International, Zhipu, OpenRouter, local Ollama, custom gateways, etc.
• fallbackModels can chain multiple providers, each with its own baseURL / model / apiKey.
• autoLocalOllama is enabled by default: it detects http://localhost:11434 at startup; if Ollama is found, it joins the fallback chain. Images stay local, and no key is required.
• API key reading order: config apiKey → environment variable VISION_API_KEY → DASHSCOPE_API_KEY. Non-anonymous entries without a key are skipped rather than failing outright.

Stability¶

• Anonymous endpoints enforce a 20-second timeout; HTTP 429 errors fail immediately without waiting for Retry-After; failed endpoints enter a 60-second cooldown.
• Without a key and no local Ollama, translation fails within seconds with actionable guidance, avoiding silent hangs.
• Errors are categorized and prompted by type: rate_limit, quota, auth, region, model_not_found, context_too_large, http.
• When the optional dependency sharp is installed, images exceeding maxImagePixels (default 4,000,000) are automatically downsampled before translation; without it, the original image is sent directly.

Installation Interaction¶

• The postinstall script asks if you have a VLM API key: answering y uses the paid fast track; answering N (default) uses the local/zero-config path; non-interactive environments skip this automatically.
• A PRIVACY NOTICE is printed at startup, indicating the currently used endpoint.

Installation and Enabling¶

The runtime requires Node >= 22.19 and dsh >= 0.1.0-rc.6. The official installation command is:

dsh plugin --profile web add dsh-vision-proxy

pnpm >= 10 intercepts dependency build scripts by default, so the first installation may exit with a non-zero code and prompt Ignored build scripts: dsh-vision-proxy, sharp. You need to approve both in the profile’s pnpm-workspace.yaml, then re-run the installation:

allowBuilds:
  dsh-vision-proxy: true
  sharp: true

dsh plugin --profile web add dsh-vision-proxy

If the official npm registry is slow, you can specify a mirror:

dsh plugin --profile web add dsh-vision-proxy --registry=https://registry.npmmirror.com

After installation, restart dsh web and select DeepSeek + Auto Vision (the deepseek-vision route) in the model selector to paste images directly in conversations.

Typical Usage¶

GUI Paste Recognition¶

1. Start DSH Web and select the deepseek-vision route.
2. Paste an image in the conversation input box and ask a question, e.g., “What do you see?”.
3. The plugin translates the image block into text (including OCR and layout description) via a VLM, and DeepSeek responds based on the transcribed text.

An example from the README: a user pastes a meme and asks a question; the VLM outputs something like “I’m a freeloader / Blue big fatty fish! …Q-version blue-haired maid outfit girl, with a blue whale tail behind, holding a bowl and chopsticks, excited expression”; DeepSeek then provides a complete visual analysis based on this. The single step takes about 7.6 seconds (README live demo data).

Custom VLM Backend¶

To use Bailian or another endpoint, override it by id in the profile; do not use insert (which would duplicate the adapter registration). Example:

# $DSH_HOME/profiles/web/cordis.patch.yml
- id: dsh-vision-proxy
  name: 'dsh-vision-proxy'
  config:
    baseURL: https://dashscope.aliyuncs.com/compatible-mode/v1
    apiKey: 'sk-…'
    model: qwen3.7-flash
    maxTokens: 4096
    timeoutMs: 120000
    maxImagePixels: 4000000
    marker: '[Image Transcription]'
    autoLocalOllama: true
    fallbackModels: []

Common backend references (all using OpenAI-compatible mode):

Use Cases and Notes¶

Who Is This For

• Users who want to process image attachments in DSH GUI using DeepSeek-V4-Pro or standard text Flash.
• Users who want images translated locally via Ollama without leaving the machine.
• Users with keys for OpenAI-compatible VLMs like Bailian, Zhipu, or OpenRouter, seeking unified integration.

When Not to Install

• If you are using the official natively multimodal model DeepSeek-V4-Flash-Vision-Exp, you can send images directly without this plugin.

Security and Permissions

• The plugin runs with the current dsh process permissions and sends image data to the configured VLM endpoint; before installation, review the source code and MIT license to ensure the endpoint and privacy policies are acceptable.
• The PRIVACY NOTICE at startup specifies the actual transcription endpoint used; under the local Ollama path, images do not leave the machine.

Configuration Notes

• For configuration overrides, use the top-level - id: dsh-vision-proxy; do not use - insert: [{id: dsh-vision-proxy, …}], as this would duplicate the instance with the same id, leading to undefined behavior.
• On Windows, environment variable changes may not take effect; it is recommended to write apiKey directly in cordis.patch.yml.
• The plugin no longer includes anonymous free endpoints as default fallbacks; if you add an anonymous endpoint yourself, you must set anonymous: true, and the 20-second timeout limit still applies.

Conclusion¶

dsh-vision-proxy connects “DeepSeek as the conversation brain” with “GUI image attachments”: even pure text DeepSeek models can now process images. The translation backend can use the official vision model, cloud VLMs like Bailian, or zero-config local Ollama. For workflows that require maintaining Pro-level reasoning in Harness without sacrificing paste-and-recognize functionality, this provides a ready-to-implement solution.

• SkillHub Directory: https://www.skillhub.cn/plugins/Flyvhidbwo/dsh-vision-proxy
• GitHub Repository: https://github.com/Flyvhidbwo/dsh-vision-proxy