Introduction

The DeepSeek Harness (DSH) agent locks a single provider/model within a session lifecycle. If the conversation history includes images while the currently selected model does not support image input, DSH’s built-in mechanisms (such as the read_image tool or ApiProxy) directly reject the processing and prompt the user to manually switch the model for the entire session.

This mandatory global switch disrupts the agent’s differentiated model choices in other parts of the session. The dsh-plugin-modality-fallback plugin aims to solve this problem, allowing the system to detect the absence of a specific modality and route only a single request to a fallback model with that modality capability, without changing the model configuration for the entire session.

Plugin Overview

dsh-plugin-modality-fallback is a DeepSeek Harness plugin that implements modality fallback routing in agent sessions.

  • Maintainer: lilei0311
  • Scope: It does not modify DSH core code and provides extension capabilities by wrapping the agent/request pipeline.

The plugin checks the session-derived message history before a request is sent. If it finds a modality not declared by the current model (currently only image supported) and a fallback route is configured, it swaps the provider and model for that request only, keeping the session’s original choice unchanged.

Core Features

  1. Modality Detection: Reads the conversation history to detect modality requirements such as images.
  2. Single-Request Routing: Only when a fallback route is configured, swaps the provider/model for requests involving the missing modality.
  3. Non-Intrusive: Runs as a standard Cordis plugin and does not touch DSH core logic.

Installation and Configuration

Install using the dsh plugin command:

dsh plugin --profile web add dsh-plugin-modality-fallback

Or install directly from the GitHub source:

dsh plugin --profile web add github:lilei0311/dsh-plugin-modality-fallback

After installation, the default configuration is an empty fallback: {}, and behavior is the same as when the plugin is not installed. Configure actual routes in cordis.patch.yml:

- id: modality-fallback
  config:
    fallback:
      image: { provider: deepseek-official, model: deepseek-vision }

If DSH is embedded in code, you can load it programmatically:

import ModalityFallback from 'dsh-plugin-modality-fallback'

await ctx.plugin(ModalityFallback, {
  fallback: {
    image: { provider: 'deepseek-official', model: 'deepseek-vision' },
  },
})

Notes and Limitations

  1. Detection Scope: Currently only the image modality is supported for detection. Extending to other modalities requires modifying the plugin code, not just the routing mechanism.
  2. Single-Request Limitation: Each request resolves at most one missing modality.
  3. Core Gating Is Not Affected: The gating of read_image and ApiProxy is not affected by this plugin; they may have already rejected the request before the plugin intervenes.
  4. Unknown Capability Handling: When model capabilities are unknown (inputModalities is undefined), the plugin treats it as “capable” and does not trigger fallback.
  5. Reasoning Overhead: Route switching discards inherited reasoning effort; fallback routes use their own adapter/provider defaults.
  6. Capability Probe Failure: If the capability probe fails, the plugin logs a warning and keeps the original routing (Fail Open), without causing the request to fail.

Conclusion

This plugin provides DSH agents with the flexibility to handle requests for specific modalities without resetting the model for the entire session. It is suitable for scenarios that require mixing capabilities from different models. See the GitHub repository for more details.