Foreword¶

Assigning the same task to coding agents, running it once yields a seemingly reasonable patch; running it independently might result in a simpler fix, better tests, or the avoidance of a subtle regression. Running multiple candidates is not difficult; the challenge lies in comparing them with the same yardstick and eliminating candidates that clearly fail tests before the comparison.

The philosophy behind DeepSeek Harness is “everything is a plugin,” and this kind of workflow orchestration is perfect for being a plugin. The dsh-llm-verifier introduced below encapsulates “Generate multiple candidates → Run tests to filter → LLM sort → Apply after human approval” into a workflow.

What is it¶

dsh-llm-verifier is a DeepSeek Harness plugin maintained by Web0926, currently in the developer preview phase with an MIT license. It runs 3 or 5 candidates for the same coding task, with each candidate working in an independent detached Git worktree; subsequently, it executes the project’s own deterministic validation first to remove failed candidates, and then uses llm-verifier to rank those that pass. Throughout the process, the original checkout remains unchanged until separate approval is granted.

Core Features¶

• Best-of-3 or Best-of-5: Defaults to 3 candidates, while high-value tasks can utilize 5.
• Validation-first selection: Runs deterministic project validation first; failed candidates will not enter the model sorting phase.
• Efficient sorting: When only 1 candidate passes, it wins by the validation result; when 2 pass, 1 pivot is used; when 3–5 pass, 2 pivots are used.
• Two approval gates: One gate before candidate execution, and another before applying the winning patch.
• Integrity check before application: Re-checks the repository path, base HEAD, clean status, and the winner-patch’s SHA-256.
• Audit-ready artifacts: The report includes ranking, changed files, time taken, process status, patch hash, verifier request, and token usage, and produces winner.patch.
• Credential control: The verification process does not receive the DeepSeek API key; logs, errors, verification output, and text diffs are desensitized based on precise credential values.
• No automatic Git changes: The plugin does not commit, push, stash, reset, nor automatically apply patches.

Validation commands support auto-detection; explicitly specified commands always take priority. When validationCommands is omitted, the plugin only accepts one identifiable root project type:

If multiple project types are hit simultaneously, or multiple JS package managers exist, or the supported types cannot be identified, the run will fail fast and require explicit command specification.

Installation and Enablement¶

Prerequisites¶

• DeepSeek Harness 0.1.0-rc.7
• Node.js 24 (engines limited to >=24 <25) and pnpm 11.7.0
• uv (Python >=3.9,<3.14 managed by uv)
• Git
• DeepSeek credentials available to Harness via credential reference DEEPSEEK_API_KEY

Build and Verify¶

The plugin is distributed via local path installation from a source build. Clone the repository, install dependencies, and run the check:

git clone https://github.com/Web0926/dsh-llm-verifier.git
cd dsh-llm-verifier

pnpm install --frozen-lockfile
uv sync --frozen --project python
pnpm run check

pnpm run check executes typecheck, test, and build sequentially; passing all indicates the local build is ready.

Add to Web profile¶

Add it to the Web profile within the plugin directory, then verify with list:

dsh plugin --profile web add "$(pwd)"
dsh plugin --profile web list

Typical Usage¶

Initiate a Best-of-3¶

Start Harness in a clean and trusted target repository:

cd /path/to/a/clean-and-trusted-git-repository
dsh --profile web

Then request using natural language, for example:

Use verified_best_of with 3 candidates to fix the login retry bug and add regression tests.
Run pnpm test for validation. Do not apply the winner yet.

The equivalent tool input is:

{
  "task": "Fix the login retry bug and add regression tests",
  "candidateCount": 3,
  "validationCommands": ["pnpm test"]
}

In the parameters of verified_best_of, task is required, candidateCount is 3 or 5 (default 3), and validationCommands uses auto-detection if omitted. The run status has three types: winner_selected, no_winner, failed. The tool returns the run ID, status, qualified candidates, ranking, report path, token usage, and (if a winner exists) the local path of winner.patch.

Apply the Winner After Review¶

After reviewing the report and patch, explicitly call apply_verified_winner:

{
  "runId": "<runId returned by verified_best_of>"
}

The plugin requests separate approval first, then applies the patch and reruns the original validation commands. Before application, it performs another integrity check on the repository path, base commit, and patch SHA-256.

Remove Plugin¶

It can be removed anytime when not in use:

dsh plugin --profile web remove dsh-llm-verifier

Applicable Scenarios and Notes¶

This plugin is suitable for repository maintainers who require high patch quality and are willing to spend more compute power to secure more stable changes, especially in scenarios where they want to systematically compare multiple candidate fixes.

Note the following points before use:

• Currently in developer preview with tight version locking: DeepSeek Harness 0.1.0-rc.7, llm-verifier 0.2.0, Node.js >=24 <25, pnpm 11.7.0, Python >=3.9,<3.14; the platform only supports macOS and Linux.
• Clear security boundaries: Public versions are only for trusted repositories. Validation commands execute target repository code on the host machine; this version is not container-bound.
• Only accepts normal, clean Git repository roots; rejects submodules, sparse checkout, linked worktree, and uncommitted changes.
• Candidate edits are located in detached worktrees under $DSH_HOME/llm-verifier/runs/<runId>; candidate Harness processes use an explicit workspace-write permission mode and do not inherit the host’s DSH_PERMISSION_MODE.
• The plugin runs with the permissions of the current dsh process; it is recommended to read the source code and license (MIT) before installation to confirm it meets your security requirements.

Summary¶

dsh-llm-verifier turns “running multiple candidates, filtering unqualified ones with tests, sorting with models, and finally manual decision” into a workflow with two approval gates; the original checkout is never automatically changed throughout, and the artifacts are auditable. The project address and listing page are as follows:

• GitHub: https://github.com/Web0926/dsh-llm-verifier
• Community directory listing: https://www.skillhub.cn/plugins/Web0926/dsh-llm-verifier

Note that skillhub.cn is a community-maintained plugin directory site and has no official affiliation with DeepSeek or High-Flyer.