Introduction

When building agents based on DeepSeek Harness (DSH), models may sometimes produce factual errors, logical contradictions, or fail to fully follow instructions. Manual review is inefficient, and relying on a model to self-correct is often unreliable. To address this issue, the plugin provides an automated solution by introducing an independent review model to audit every assistant response and prompt the agent to repair it when problems are found.

Plugin Introduction

dsh-quality-review is a workflow plugin for DeepSeek Harness (DSH). It is maintained by CAI-MH, uses an independent review model to audit every assistant response, and prompts the agent to repair it when problems are found, supporting up to 2 rounds of correction.

Core Features

The plugin provides the following core capabilities:

  1. Audit by an Independent Review Model: At the agent/turn-stopping point near the end of each conversation turn, the plugin captures the assistant’s latest visible response and sends it to an independent review model for audit.
  2. Guided Repair: When the review result is determined to be “failed”, the plugin sends the agent a specific repair request, forcing the model to rewrite the response and keeping the conversation turn open until the requirements are met.
  3. Multi-Dimensional Review Criteria: Supports enabling or disabling the following review aspects according to configuration:
    • Factual accuracy
    • Completeness
    • Logical consistency
    • Instruction following
  4. SOP Reference Standards: Supports loading contents from an SOP folder as quality standards. When the user prompt matches a file name in the folder, the file contents are injected into the review prompt, adding a review aspect of “compliance with SOP rules.”

Installation and Activation

The installation process requires using the plugin management command provided by DSH.

dsh plugin --profile web add dsh-quality-review

After installation, the plugin runs with default configuration. It can also be enabled by modifying the dependencies in package.json and restarting dsh web.

Configuration and Usage

Every field has a safe default value, so the plugin can be inserted without configuration. To customize the review logic, the relevant configuration items need to be overridden in the agent configuration file cordis.patch.yml.

- id: quality-review
  config:
    enabled: true            # 总开关；设为 false 时插件加载但不生效
    reviewer:
      provider: ''           # 审查模型提供商 ID；留空则复用代理自身的 provider
      model: ''              # 审查模型 ID；留空则复用代理自身的模型
    maxRounds: 2             # 每轮对话允许的最大引导修复轮次（循环保护，最大为 5）
    aspects:
      factualAccuracy: true
      completeness: true
      logicalConsistency: true
      instructionFollowing: true
    minReplyChars: 200       # 跳过短于该字符数的回复
    sop:
      enabled: true          # SOP 参考标准开关（默认开启）
      dir: ''                # SOP 文件夹路径；留空则使用 DSH_HOME/quality-review/sop
    exemptPatterns:          # 额外的静态关键词列表；匹配时跳过审查（可选）
      - 代码 review

SOP Folder Configuration

The SOP folder is located by default at DSH_HOME/quality-review/sop. The plugin automatically scans this folder. In each conversation turn, as long as the user prompt contains any file name (case-insensitive, ignoring the extension), the plugin loads the contents of all files in the folder and injects them into the review prompt as “task standards.” The review model evaluates whether the response complies with these standards.

Applicable Scenarios and Notes

This plugin is suitable for agent scenarios requiring highly reliable output, especially tasks involving factual retrieval, logical reasoning, or strict format requirements.

Notes:
1. The plugin runs with the permissions of the current DSH process. It is recommended to inspect the source code before installation.
2. The plugin is released under the MIT license.

Conclusion

By following the steps above, you can add a layer of automated quality gating for agents in DSH. Through independent model auditing and round limits, it effectively reduces model hallucinations and missed instructions, improving the stability of agent responses.

  • Plugin Catalog: https://www.skillhub.cn/plugins/CAI-MH/dsh-quality-review
  • GitHub Repository: https://github.com/CAI-MH/dsh-quality-review