Introduction¶

When running tasks on macOS using DeepSeek Harness (DSH), there is an unavoidable limitation: the connected model is a pure text model that cannot “see” anything on its own. To have it read an error message from a screenshot or recognize an image in the clipboard, human intervention is required to convert the content to text first.

There are two common workarounds: connecting to a third-party multimodal API or downloading an open-source visual model locally. The former requires obtaining an API key and paying per use, while the latter requires downloading and maintaining model weights. dsh-mac-vision offers a third option—directly invoking macOS’s built-in Apple Vision to perform OCR and visual detection locally, without needing an API key or downloading third-party models.

DSH’s philosophy is “everything is a plugin,” and visual capabilities can be made into one too. Below, we introduce its positioning, capabilities, and integration methods.

What is it¶

dsh-mac-vision is a native macOS vision plugin maintained by Kevoyuan. It provides local OCR and visual detection capabilities for images, the clipboard, the screen, and application windows to pure text models via Apple Vision. It is open-sourced under the MIT License.

The plugin consists of two parts internally:

Vision Tools: Actually perform screenshots, OCR, and Apple Vision detection.
Built-in Skill: Responsible only for instructing the model when to call vision tools and how to express observations and inferences; it cannot execute OCR independently.

When installing the plugin, both parts are installed together. There is no need to choose between a Plugin and a Skill, nor is there any need to download, copy, or install SKILL.md separately.

Core Capabilities¶

Read local images, clipboard images, full screen, foreground window, or specific windows.
Use macOS’s built-in Apple Vision to perform OCR locally, without downloading third-party visual models, API keys, or incurring pay-per-use fees.
Return structured evidence such as text, confidence scores, coordinates, candidate results, and re-inspection status.
Automatically crop and scale small-font or formula areas for secondary OCR.
Optionally run image classification, barcode detection, salient region detection, and people/pet detection.
Distinguish between direct observations, semantic inferences, and uncertain results.
Support Harness cancellation signals, timeouts, and output size limits.

The plugin registers two tools for the model:

mac_vision_inspect: Analyze images, clipboard, screen, or windows.
mac_vision_list_windows: List visible windows for easier selection of specific windows.

Installation and Usage¶

Prerequisites for installation:

macOS
Node.js 22 or higher
DeepSeek Harness with dsh installed and running
Xcode Command Line Tools, or Xcode containing a Swift compiler

If you don’t have a Swift compiler yet, run:

xcode-select --install

Then install the plugin:

dsh plugin --profile default add dsh-mac-vision

This command installs both the vision tool and its model usage strategy. After installation, verify the configuration first, then start:

dsh --profile default --dump-config
dsh --profile default

The first time a vision task is executed, the plugin uses the system’s Swift compiler to build a local helper and caches it, after which it can be used directly.

Regarding permissions: reading local images does not require screen recording permissions. When reading the screen or application windows, macOS may require authorization for the terminal or host application running DSH in “System Settings → Privacy & Security → Screen Recording”. After authorization, the application may need to be restarted.

Typical Usage¶

After the steps above, simply describe the task to the model; manual tool invocation is usually not required. Below are examples from the README:

Read the title and main content from this image: /absolute/path/to/slide.png

Look at what is currently displayed in the foreground window and distinguish between what you see directly and your inferences.

Read the text from the clipboard image and clearly mark any content that cannot be confirmed.

List the current windows, then check for error messages in the browser window.

Check if this screenshot contains a QR code and extract the visible text: /absolute/path/to/image.png

The results returned by the model are distinguished into three layers:

Observed / Direct Observation: Evidence directly returned by OCR or completed detectors.
Inferred / Inference: Explanations made based on layout, text position, or multiple observations.
Uncertain / Uncertainty: Conflicting OCR, unverified formulas, or detectors that were not run.

If OCR of small text or formulas is inaccurate, you can ask the model to re-examine specific areas and explicitly request that uncertain items be retained.

Configuration¶

Most users do not need to modify the configuration. The default values are located in the plugin’s cordis.patch.yml:

config:
  timeoutMs: 45000
  maxOutputBytes: 8388608
  defaultMode: fast
  defaultLanguages: []
  defaultRefineText: true
  defaultRefineLimit: 12
  defaultRefineScale: 3
  allowedSources: [file, clipboard, screen, front-window, window]

You can override settings in the profile’s cordis.patch.yml. Note that Harness patches replace the entire config rather than deep merging it item by item; therefore, when overriding, you need to rewrite the entire configuration block. For example, to only allow reading local files:

allowedSources: [file]

Updates and Uninstallation¶

Update to the latest version:

dsh plugin --profile default update dsh-mac-vision

Uninstall:

dsh plugin --profile default remove dsh-mac-vision

If you are using a different profile, simply replace default in the command with the corresponding name.

Suitable Scenarios and Notes¶

Suitable users:

Using DSH on macOS and want the model to read images, clipboard, screen, and windows.
Do not want to apply for third-party OCR / visual APIs, nor download and maintain visual models.
Concerned about data privacy: screenshots, OCR, and visual detection are all performed locally, and images are not sent to third-party OCR or visual services.

Notes before use:

The plugin runs with the permissions of the current dsh process; any local files, clipboard, or screen content it can access is constrained by these permissions. Before installation, it is recommended to check the plugin source code and license (MIT) to confirm it meets your security requirements.
When you need to read the screen or windows, complete the “Screen Recording” authorization mentioned above first.
The plugin’s vision capabilities are free and performed locally; whether the language model and its service used by DeepSeek Harness are paid depends on your own Harness configuration and is unrelated to this plugin.

Summary¶

For DSH users, dsh-mac-vision solves a very specific problem: using a single installation command to grant pure text models the ability to read images and screens locally on macOS, without API keys, per-use fees, or images leaving the local machine.

dsh-mac-vision: Enable DeepSeek Harness to perform OCR and visual detection locally on macOS