Introduction¶

When building agents using DeepSeek Harness (DSH), you often encounter the same limitation: the text model at hand lacks native image input capabilities. Screenshots or data charts pasted into the conversation cannot be read by the model. Without a plugin, you can only use a vision model upstream to read the image first, then manually paste the description back into the conversation, breaking the chain in the middle.

DSH’s philosophy is “Everything is a plugin,” and such gaps are well-suited to be filled by plugins. The dsh-tool-visual-primitives introduced below is one such plugin: it delegates the image to an external vision model for analysis, returning pure text visual evidence with spatial location information, and then hands it back to the original conversation model for continued reasoning. The enhanced model does not require native image input capabilities.

What is it¶

dsh-tool-visual-primitives is a DSH plugin maintained by InkshadeWoods, with source code hosted on GitHub. The current version is 1.4.0 and the license is MIT. The core concept draws inspiration from DeepSeek’s “Thinking with Visual Primitives”: it transforms image understanding into evidence that is usable and verifiable for subsequent reasoning by utilizing normalized coordinates and referenceable objects.

The plugin has two entry points that share the same vision_analyze analysis core: explicit tool calls, and the [vision] model variant within the conversation.

How it works¶

The flow of a request inside the plugin is as follows:

Image + User Question
  → detectVisionMode() → shouldUsePrimitives() → buildVisionPrompt()
  → External Vision Model (OpenAI Compatible Chat Completions)
  → Pure Text Visual Evidence (may include coordinate-based primitives)
  → Original Text Model continues answering

Three control dimensions are mutually orthogonal: Mode determines the task type, Detail determines the information density, and Primitives determines whether to force structured spatial evidence.

The plugin automatically recognizes 11 categories of visual tasks: caption, object_inventory, multi_subject, counting, grounding, spatial_relation, comparison, path_tracing, topology, ui_analysis, and document_visual. The keyword with the highest priority determines the Mode; if no match is found, it falls back to caption. For example, “How many buttons are on this interface?” will be identified as ui_analysis.

Core Features¶

• Two entry points share the vision_analyze core: explicit tool calls and the [vision] model variant in the conversation.
• Three levels of analysis detail: brief, standard (default), and verbose.
• Three visual primitive strategies: auto (default), on, and off. Primitives use <ref>, <box>, and <point>, with coordinates ranging from 0 to 999.
• Only appends the [vision] variant to selected text models; the original model remains unchanged.
• Session-level visual evidence cache: follow-up questions reuse cached evidence only if it covers the new issue; otherwise, the image is re-analyzed.
• Native settings page: secure key storage, connection testing, /models search selection, manual model ID, and collapsible selection of target models to enhance by vendor.
• Supports OpenAI-compatible endpoints: POST <Base URL>/chat/completions and optional model directory GET <Base URL>/models; Xiaomi Mimo URL automatically uses the api-key header.
• Concurrent analysis of the same image and prompt is automatically merged into a single upstream request. Diagnostic log settings are disabled by default (introduced in 1.3.0).
• Starting from 1.4.0, the “Conversation Vision Model” list in the settings page uses the new model directory interface from DSH 0.1.2-rc.1.

Installation and Enabling¶

Prerequisites¶

• A running DSH Web Profile. The current version requires DSH 0.1.2-rc.1 or higher; engine declarations in dsh.plugin.json and package.json are synchronized and tightened.
• Node.js >= 20 and pnpm.
• An accessible vision model service, defaulting to an OpenAI-compatible endpoint. The vision service and the text model to be enhanced can be from different vendors.

One-click Installation¶

Execute the following in PowerShell or a terminal (assuming DSH is globally installed, i.e., npm install -g @deepseek-ai/dsh):

dsh plugin --profile web add dsh-tool-visual-primitives@latest

If you are not using the web Profile, change --profile to the corresponding name. Upon successful installation, DSH will recognize the dsh.bundle.patch declaration within the package and automatically add the plugin to dsh.profile.bundles; this process does not directly modify the Profile’s own cordis.patch.yml, allowing for coexistence with plugins like dsh-better-sidebar.

You need to fully restart DSH after installation. When updating the client interface for the first time, force a refresh in the browser by pressing Ctrl+Shift+R.

First-time Configuration¶

Open DSH Settings → Visual Analysis and proceed in order:

1. Fill in API Key, Base URL, and the vision model.
2. Click “Load Models”; the plugin fetches a searchable list from <Base URL>/models. If the service does not provide a model directory, fill in the custom model ID directly.
3. Click “Test Connection”.
4. In “Conversation Vision Model”, check the text models you wish to append [vision] to.

Two details: The saved API Key will not be echoed when reopening the page; entering a new value overwrites the old value, and it is only deleted when you click “Clear API Key”. When troubleshooting, you can enable the diagnostic log (default is off); the plugin will output runtime logs to the DSH console.

Local Mount from Source Code¶

For development and debugging, you can use the GitHub source code:

$source = 'D:\DSH\dsh-tool-visual-primitives'
git clone https://github.com/InkshadeWoods/dsh-tool-visual-primitives.git $source

Set-Location $source
pnpm install
pnpm run build

dsh plugin --profile web add $source

Uninstall¶

First, click “Clear API Key” in the settings page, then execute:

dsh plugin --profile web remove dsh-tool-visual-primitives

Typical Usage¶

Using [vision] in Conversation¶

After completing the configuration steps above:

1. Check a text model in the “Conversation Vision Model” in plugin settings.
2. Reopen the conversation model list and select the newly added “Model Name [vision]”.
3. Upload, paste, or drag and drop an image and ask a question directly.

The plugin only replaces the image block with visual evidence text; the final answer is still generated by the selected original text model. For follow-up questions explicitly pointing to the most recent image, the plugin checks if the cached evidence covers the new task, details, and focus objects; if coverage is insufficient, the image is re-analyzed rather than treating an insufficient old answer as fact.

Explicitly Calling vision_analyze¶

The tool accepts and only accepts one image source: a local absolute path or an HTTP(S) URL.

{
  "image_path": "D:/images/dashboard.png",
  "prompt": "统计界面上可点击的主要按钮，并标出它们的位置"
}

{
  "url": "https://example.com/chart.png",
  "prompt": "解读图表的趋势，并说明读不清的标签"
}

When visual primitives are enabled, the returned evidence is organized by headings such as [Mode], [Visual Primitives], [Observations], [Relations], [Uncertainty], and [Answer]. Location information takes the form of:

<ref>submit_button</ref><box>[[742, 861, 900, 930]]</box>
<point>[[125, 430], [210, 430], [300, 510]]</point>

All coordinates are relative to 0–999, not the original image pixels.

Analysis Parameters¶

After the connection configuration is complete, adjust parameters as needed. The settings page provides these items:

Suitable Scenarios and Considerations¶

Suitable Scenarios:

• The conversation model lacks native image input capabilities, but the task involves reading screenshots, interpreting charts, or replicating UI.
• You want the vision service and the text model to belong to different vendors.

The README includes end-to-end test records (conversation image understanding, screenshot-driven UI replication), with assets and results saved in the test/ directory of the repository. The generated results depend on the external vision model, text model, prompt, and image quality.

Pre-use Considerations:

• Security restrictions: Remote URLs cannot point to localhost, private network addresses, or contain username/password; redirects will be rejected to reduce SSRF risk.
• Permissions: The plugin runs with the current dsh process permissions. Please review the source code and license yourself before installing.
• Uninstall: Clear the API Key in the settings page before uninstalling, then execute the uninstall command.

Related Links¶

dsh-tool-visual-primitives offloads “reading images” to an external vision model, allowing the text model to obtain visual evidence that is coordinate-based and verifiable. For developers building agents with DSH who are stuck due to the model’s limited visual capabilities, this is a patch for a specific gap.

• GitHub Repository: https://github.com/InkshadeWoods/dsh-tool-visual-primitives
• Community Plugin Directory Entry: https://www.skillhub.cn/plugins/InkshadeWoods/dsh-tool-visual-primitives

The directory site is independently maintained by the community and has no official affiliation with DeepSeek / Hypothesis.