Preface¶

The core philosophy of DeepSeek Harness (dsh) is “Everything is a plugin”: models, tools, sessions, sandboxes, and interfaces can all be swapped out. The official repository is deepseek-ai/deepseek-harness. It is usually powered by pure-text models such as DeepSeek, dense Qwen, Llama, and Mistral in daily use.

Two obstacles arise when pasting screenshots into conversations. The first occurs before sending: dsh checks attachments against the modality declared by the route, and text models will be rejected directly, with the model name pointed out. The second is more troublesome: if you hardcode input: [text, image] on the text route, the attachment can be sent, but the upstream will return 400 "not a multimodal model" mid-round—at which point the user’s message has been saved to disk, and the session will repeatedly retry a request that will never succeed.

Adding just a “image viewing tool” cannot unlock the first barrier: the model must first see the image before deciding to call the tool, which is exactly the request that dsh rejects. Community maintainer tonyd2wild thus created DeepSeek-Harness-Vision-Tools: the text model continues to act as the brain, the vision model only handles seeing, and image bytes are not injected into the brain’s context. This article is organized after cross-checking with the plugin directory page, GitHub README, examples/dsh.md, and the repository source code.

What is this¶

DeepSeek-Harness-Vision-Tools is a session and message community plugin maintained by tonyd2wild under the MIT license. Both the directory page and GitHub list 10 stars; the directory收录 date is 2026-08-10, and the latest push to the repository was 2026-08-13. The plugin package name is dsh-plugin-vision, and the version in package.json is 0.1.0.

The opening of the README clearly states: this is an unofficial community project, not affiliated with, endorsed by, or maintained by DeepSeek AI. Please submit issues to this repository, not to DeepSeek. The community plugin directory deepseek-harness-plugin.com is an independent site and is not the official app store for DeepSeek / HyperFiction.

The problem it solves can be condensed into one sentence: any text model paired with any vision model, providing two image access channels for dsh.

The two channels are complementary, not redundant. The tool cannot handle chat attachments: the attachment must first reach the model before the model will call the tool. The proxy cannot pick up disk files that the agent has not yet included in the message. Images are only sent to the vision model; the brain sees text like [Image: ...].

Core Features¶

Separate brain and eyes, both customizable¶

The repository deliberately does not hardcode models. The maintainer’s own DeepSeek build and hardware combination is difficult for others to reuse, so the recipe only leaves two slots:

The same set of eyes can serve both channels. The tool selects the backend based on the call; the proxy specifies one via --vision-model:

The fast default given in MODELS.md is Qwen3.5-0.8B: Apple Silicon uses MLX (mlx-community/Qwen3.5-0.8B-MLX-8bit), Windows / Linux / NVIDIA uses llama.cpp (GGUF + mmproj). It occupies approximately 2–3 GB of Mac unified memory, or about 2 GB of VRAM on a PC. You do not need to enable both tiers; start with fast.

Ollama currently cannot load Qwen3.5’s standalone mmproj, so the default fast model cannot run via Ollama. If you insist on using a one-line Ollama command, the repository recommends switching to Moondream or Qwen2.5-VL, then pointing the backend to http://127.0.0.1:11434/v1/chat/completions.

Vision proxy: turn chat attachments into text before they reach the brain¶

The proxy is a local HTTP service that only depends on Python’s standard library, pretends to be an OpenAI API, and sits between dsh and the text upstream:
1. POST /v1/chat/completions: Replace each image_url block in every message with {"type":"text","text":"[Image: ...]"}; if the entire message becomes text, consolidate it into a plain string (some servers are more picky about block arrays).
2. Forward GET /v1/models and other GET requests as-is to facilitate dsh model discovery.
3. Forward streaming responses byte-by-byte. dsh streams output each round, and buffering would freeze the interface.
4. Degrade gracefully on vision failures without throwing fatal errors: blocks become [Image: (image could not be analyzed: ...)], and the round can still complete.
5. The request body has a 64 MB limit because images are inline base64.

dsh’s model route must point to the proxy and declare input: [text, image]. This statement is true for the proxy, but false for the text brain. The route describes the object it is conversing with.

analyze_image: let the agent decide when to view disk files¶

plugin/vision/index.js uses @deepseek-ai/dsh-tools’s defineTool() to register a tool visible to the model. The parameters come from the source code:

The tool reads the file as base64, POSTs it to the corresponding backend’s /v1/chat/completions, and writes the returned plain text into the tool result. An unknown backend will throw an error and list valid names, without silent fallback, to avoid missing the mistake when typing detailed as fast.

Reading files prioritizes ctx.fs (compliant with workspace boundaries and approval policies). If it falls back to readFileSync, the sandbox is bypassed, and the model could theoretically read any file on the disk via this tool. The repository requires: only rely on this fallback on trusted, attended machines, and lock it down before unattended operation.

Installation and Enablement¶

The installation command given on the plugin directory page is as follows, executed in the DeepSeek Harness terminal:

dsh plugin add github:tonyd2wild/DeepSeek-Harness-Vision-Tools

The directory page also provides a reproducible method: append the commit hash after the repository name.

dsh plugin add github:tonyd2wild/DeepSeek-Harness-Vision-Tools#commit

The plugin runs with the permissions of the current dsh process, and may execute code during installation. You should check the source code repository and license before installing.

This repository is not as simple as “just dropping a JS plugin”: the proxy is an independent Python process, and the tool is in plugin/vision/. For both channels to work, you need to start the eyes service, proxy, and tool separately according to the README. Prerequisites (from README “What you need”):
- A text model already running on dsh (any OpenAI-compatible endpoint)
- A local VLM; the minimal fast tier requires approximately 2–3 GB of additional resources
- Python 3.8+ (the proxy only uses the standard library, no third-party dependencies)
- pnpm (called by dsh plugin; required only when installing the tool)
- Vision runtime: MLX (mlx-vlm) for Mac, llama.cpp (or Ollama for fallback models) for Windows / PC

Quick start:

git clone https://github.com/tonyd2wild/DeepSeek-Harness-Vision-Tools
cd DeepSeek-Harness-Vision-Tools
cp .env.example .env      # Change the endpoint to your own host
./setup.sh                # Start the local vision service (starts the proxy together when RUN_PROXY=1)

Channel 1, Proxy (chat attachments):

python3 shim/vision_shim.py --port 8900 \
  --upstream http://127.0.0.1:8000 \
  --vision-url http://YOUR_FAST_VISION_HOST:8081/v1/chat/completions \
  --vision-model your-fast-vlm

Environment variables correspond one-to-one with CLI arguments, with CLI arguments taking precedence: SHIM_PORT / --port, VISION_TARGET / --upstream, EYES_URL / --vision-url, EYES_MODEL / --vision-model. Health check:

curl http://127.0.0.1:8900/health

Channel 2, Tool (disk files). First copy the plugin to a stable path, and follow the repository’s “Trap 2” to modify the link: for @deepseek-ai/dsh-tools in package.json to point to the version bundled with the current Harness, do not install the old package from npm (the documentation states that the npm version 0.0.1-rc.1 is older than the bundled 0.1.0-rc.6, and it will reference an unpublished package):

cp -r plugin/vision ~/.dsh/plugins/vision
dsh plugin --profile <p> add link:~/.dsh/plugins/vision

Replace <p> with the actual profile, such as web or headless. Plugins are parsed from the profile directory, do not drop them into the installation directory’s node_modules, otherwise all profile startups will throw ERR_MODULE_NOT_FOUND. You need to add this for every profile you want to use.

Typical Usage¶

Point a dsh route to the proxy¶

The model route is written in the pi-ai provider block in $DSH_HOME/settings.yaml. baseURL points to the proxy, and declares text-image input. Below is the placeholder configuration from examples/dsh.md, replace the host and model id with your own:

llm-pi-ai:
  providers:
    vision-proxy:
      displayName: Your Text Model (via vision proxy)
      apiKeyEnv: YOUR_PLACEHOLDER_KEY_ENV
      api: openai-completions
      baseURL: http://127.0.0.1:8900/v1
      models:
        - id: your-text-model-id
          contextWindow: 262144
          maxTokens: 32768
          input: [text, image]

You also need to fill in apiKeyEnv for upstream without an API key, pointing to any non-empty environment variable. If omitted, pi-ai will attempt environment discovery, and fail with PI_AI_ERROR: No API key if not found.

Add a backup route that connects directly to the upstream without declaring input so you can continue working if the proxy goes down:

    text-only-direct:
      displayName: Your Text Model (direct, no vision)
      apiKeyEnv: YOUR_PLACEHOLDER_KEY_ENV
      api: openai-completions
      baseURL: http://127.0.0.1:8000/v1
      models:
        - id: your-text-model-id
          contextWindow: 262144
          maxTokens: 32768

Model routes hot-reload without restarting. After saving, select the proxy route in the model selector and attach an image to test.

Configure two backends for analyze_image¶

Write in the plugin configuration block or environment variables:

export VISION_FAST_URL=http://YOUR_FAST_VISION_HOST:8081/v1/chat/completions
export VISION_FAST_MODEL=your-fast-vlm
export VISION_DETAILED_URL=http://YOUR_DETAILED_VISION_HOST:8010/v1/chat/completions
export VISION_DETAILED_MODEL=your-detailed-vlm

The agent calls the tool as analyze_image(path, backend, prompt). For example, override the prompt for screenshots or receipts: prompt: "List every object and any visible text.".

On the web interface, model-facing tool rows are disabled at the host level, and you need to rely on agent preset to see them. Do not modify the standard preset distributed with the installation, as it will be overwritten on upgrade. Create a user preset with a new id (for example, standard-vision), combining web and analyze_image. User presets cannot reuse already distributed ids, otherwise they will be silently overridden. Then set the default in settings.yaml (hot-reloadable):

agent-presets:
  default: standard-vision

Presets are only mounted when a session starts for the first time. Checking the clean startup log will not prove the tool is active; you need to start an actual session.

Tell the agent: it has not become a multimodal model¶

dsh reads $DSH_HOME/AGENTS.md into every session. After the proxy is active, the model often infers that it has been switched to a vision route. The repository provides a pasteable explanation, the core of which is: you are still the original text model; images have been converted to [Image: ...] by the proxy or tool; please say “the description shows…”, do not say “I can see…”.

Repository’s end-to-end verification method¶

When a text model receives an image block, it will generate a seemingly reasonable description that sounds like success. The repository used solid-color images that the model could not guess for testing:
- Non-streaming: solid green → “The image is a solid, bright green.”
- Streaming: solid blue → “a solid, uniform field of deep, saturated blue” (5 chunks, clean [DONE])
- The same text model will return 400 "is not a multimodal model" when directly fed images, proving that the text came from the proxy’s vision layer, not the brain

Reproduction method: generate a few solid-color PNGs, attach one in a chat and ask about its color (proxy path), or have analyze_image point to the file (tool path), and verify that the description matches.

Applicable Scenarios and Notes¶

Suitable for: people who want to keep their existing text brain while giving dsh contextual awareness of screenshots, photos, and camera frames. The repository’s original wording is: it adds capabilities without taking over the deployment; the proxy is an independent process that can be killed at any time; the tool is just an additional capability that the agent can call.

Not suitable as a native multimodal solution. The repository’s “Honest limits” section is straightforward:
- Descriptions are lossy. Details, precise spatial relationships, and small objects may be lost. For layout reasoning, switch to the larger detailed model.
- Small VLMs have weak performance when reading text in images. fast trades size for accuracy. For OCR-heavy tasks, point the proxy or tool to a larger VLM, and accept the additional memory and latency.
- The brain reasons over text, not pixels. Most agent tasks (what’s on the screen, what’s in the photo, reading an error message) are sufficient; critical OCR or fine-grained visual reasoning is not.
- input: [text, image] is a declaration, not a check. Only declare it for endpoints that actually accept images: the proxy, or a true multimodal VLM. Declaring it on a pure text model will result in a mid-round 400 after the message is saved to disk.

A few more pitfalls from examples/dsh.md:
1. The llama.cpp path must include --mmproj (vision projector). backends/pc_llamacpp.sh will pass this automatically. If omitted, llama.cpp will silently run in pure text mode, with descriptions being empty空话. This is the most common “process is running but cannot see images” issue.
2. Windows absolute paths work in presets, but in profile patches, C: will be treated as a URL scheme, throwing ERR_UNSUPPORTED_ESM_URL_SCHEME. Using bare package names is more stable on both sides.
3. Modifying settings.yaml (model routes, default preset) does not require a restart; modifying the preset’s agent.cordis.yml also does not, the next session will use the file’s timestamp to update; modifying the plugin’s index.js requires a restart (