Introduction

DeepSeek Harness (DSH) adopts a pluggable architecture, allowing dynamic routing to specific capabilities through natural language requests. Medical imaging analysis involves multiple tasks such as report generation, lesion localization, segmentation, and classification, and different modalities (X-ray, CT, MRI, etc.) require different processing tools. dsh-medomni is a DeepSeek Harness plugin maintained by bowang-lab. It implements the MedOmni strategy and provides language model agents with a set of modality-specific tools, enabling them to automatically choose the appropriate inference workflow based on natural language requests.

Plugin Positioning and Capabilities

Plugin Name: dsh-medomni

Core Positioning: A DeepSeek Harness plugin for medical imaging analysis.

Main Capabilities:
* Report Generation: Supports candidate report generation for chest X-ray, CT, MRI, and retinal images, including comparison between current and prior images.
* Localization and Annotation: Uses evidence boxes to localize suspected X-ray lesions and annotates normal anatomical structures.
* Segmentation and Masking: Supports generating segmentation masks and overlays for X-ray, ultrasound, retinal images, as well as CT/MRI NIfTI volumetric data; supports fixed-label anatomical segmentation for CT/MRI.
* Classification: Classifies ultrasound images.
* Intelligent Routing: Parses intent from natural language requests and automatically selects report, classification, localization, or segmentation tools.
* Input Support: Supports 2D/3D imaging input via file paths (3D supports NIfTI and DICOM directories), and also supports pasting 2D images via image-enabled Provider routes.

Installation and Activation

1. Install the Plugin

In a configured DSH environment, install the plugin with the following command. This command adds the plugin to the profile’s package.json and automatically mounts cordis.patch.yml.

dsh plugin --profile web add github:bowang-lab/dsh-medomni

After installation, restart DSH (or the Web session) to load the new plugin.

2. Prepare the Environment

After installation, you need to initialize the plugin’s shared Python environment. Run the following command:

cd ~/.dsh/profiles/web
./node_modules/.bin/dsh-medomni setup

This step creates a Python virtual environment and installs general dependencies. Model checkpoints and some additional dependencies (such as BiomedParse’s detectron2) are downloaded on demand when used for the first time.

3. Configure Prerequisites

Model inference runs locally. The plugin needs access to Hugging Face to download model checkpoints. For restricted models (such as MAIRA-2, BiomedParse, and MedGemma), you must accept the agreements on Hugging Face and set the HF_TOKEN environment variable.

An NVIDIA GPU (CUDA) is recommended. Although some tools support CPU mode, multi-GB visual language and segmentation models are less efficient on CPU.

Usage Examples

After the plugin is installed and initialized, you can interact with DSH directly using natural language. For example:

“Generate a radiology report for this chest X-ray: /path/to/chest_xray.png“

The system automatically identifies the modality (chest X-ray) and intent (report generation) in the request and invokes the corresponding tool.

Handling Pasted Images

If you need to paste an image (2D medical imaging) from the clipboard, make sure the selected Provider route supports image input.

  1. In the model selector in the chat interface (usually at the bottom-right), select the item labeled "+ dsh-medomni Vision".
  2. After switching to that route, paste the image.

Notes

  • Local Execution: All model inference is executed on the configured local machine and does not involve uploading data to the cloud.
  • File System: Input files and generated preview images are read from and written to the local file system. File retention policy is controlled by the DSH session.
  • Hugging Face Token: HF_TOKEN is only used to access restricted models. It is not stored by the plugin but read from the process environment.
  • Version Requirements: The plugin depends on Node.js >= 22.

Summary

dsh-medomni encapsulates complex medical imaging analysis workflows as a DSH plugin and simplifies the invocation chain from text to specific medical tools through the Agent routing mechanism. It is suitable for developers who need to deploy in a local environment and want to leverage the DeepSeek Harness ecosystem for multimodal task orchestration.

  • GitHub Repository: https://github.com/bowang-lab/dsh-medomni
  • Plugin Directory: https://www.skillhub.cn/plugins/bowang-lab/dsh-medomni