Introduction

In DeepSeek Harness (DSH), when an agent runs a long-running task and declares it “complete,” that is usually only a self-declaration based on its own judgment. Models may close the goal before tests pass or try to end the turn by skipping verification.

The dsh-verify-judge plugin intercepts update_goal(complete) calls and turns “completion” into a conditional action: before a goal can be closed, the project’s test suite must be run first. If tests fail, the plugin rejects the goal closure and feeds the failure information back to the agent. It also checks and forces re-verification at turn boundaries, ensuring the agent continues working until the task is truly complete.

Plugin Purpose

This is a verification plugin for DeepSeek Harness, maintained by zriyox. It verifies whether an agent has truly completed the task by running the workspace’s verification commands (usually the test suite), rather than relying on the agent merely declaring completion in the log.

Core Features

The plugin mainly provides the following capabilities:

  1. Interception and Verification: Intercepts update_goal(complete) calls and runs the project’s test suite before the goal is closed.
  2. Turn Boundary Check: Checks and forces re-verification at the end of a turn, preventing the agent from skipping tests by merely saying “complete” in the chat.
  3. Automatic Command Detection: Automatically detects test commands.
    • scripts.test in package.json.
    • A go.mod file (runs go test ./...).
    • A Cargo.toml file (runs cargo test).
  4. Custom Configuration: Supports configuring custom verification commands.
  5. Undetected Command Handling: When no command is detected, supports an allow (default) or deny policy.
  6. Timeout and Feedback: Supports command timeout control and captures the tail of the output as feedback on failure.

Installation and Enablement

Install the plugin using the official command. After installation, restart dsh web or wait for the patch to reload.

dsh plugin --profile web add github:zriyox/dsh-verify-judge

Configuration

Add the following to the plugin’s configuration entry (profile or home cordis.patch.yml):

- id: verify-judge
  config:
    commands: ['pnpm test', 'pnpm run lint']  # 指定运行命令
    onUndetected: deny                        # 未找到命令时拒绝完成
    timeoutMs: 300000                        # 单个命令超时时间
    outputTailChars: 4000                    # 失败输出截取字符数
    gateTurnEnd: true                         # 在回合结束时重新验证
    maxTurnEndRetries: 2                      # 每回合最大重试次数

Configuration Items

  • commands: Explicitly specified verification commands, taking priority over automatic detection. They must run in the session workspace.
  • onUndetected: The policy when no command is detected; defaults to allow.
  • timeoutMs: Timeout limit for an individual verification command.
  • outputTailChars: Maximum number of output tail characters sent to the agent when verification fails.
  • gateTurnEnd: Whether to trigger re-verification at the end of a turn when a goal is active.
  • maxTurnEndRetries: Maximum number of times the agent is forced to continue working within a turn due to verification failures.

Dependencies and Requirements

The plugin requires the following dependencies to run:

  • Node.js Version: ^22.19.0 or >=24.0.0
  • Core Dependencies:
    • @deepseek-ai/cordis (^4.0.1)
    • @deepseek-ai/dsh-agent (^0.1.0-rc.6)
    • @deepseek-ai/dsh-goal (^0.0.1-rc.1)
    • @deepseek-ai/dsh-llm (^0.0.1-rc.1)
    • @deepseek-ai/dsh-session (^0.0.1-rc.1)
    • @deepseek-ai/dsh-shell (^0.1.0-rc.5)
    • @deepseek-ai/dsh-tools (^0.0.1-rc.1)
  • Development Dependencies: @types/node (^22.20.0), typescript (^5.9.3)

The plugin intercepts calls via tools/pre-execute waterfall and does not modify non-update_goal calls.

Use Cases

This plugin is suitable for scenarios that require strict verification of agent outcomes, especially when using DSH’s autonomous goal loop. It ensures that the agent cannot end a task by “self-declaration” alone and must pass real code validation (such as passing tests).

Summary

dsh-verify-judge shifts agents from “self-declaring completion” to “proving completion.” By running the test suite and enforcing turn-boundary checks, it provides a mandatory verification mechanism. When properly configured, the plugin is transparent to honest workflows and only intervenes when verification fails.