Introduction

DSH emphasizes “everything is a plugin.” In agent development, determining whether a candidate output satisfies expected fragments often requires a repeatable judgment within CI. aryswisnu/dsh-eval-regression provides a deterministic, CI-safe golden-output evaluation tool for comparing required fragments and forbidden fragments in candidate outputs.

It is not a model quality review, nor does it judge semantic correctness. Its goal is narrower: to provide repeatable pass/fail evidence.

What is this

aryswisnu/dsh-eval-regression is a DeepSeek Harness (DSH) plugin, positioned as a deterministic, CI-safe golden-output evaluation. The repository owner is aryswisnu, and the license is MIT.

It registers a model-callable tool:

evaluate_golden_output

This tool is used to compare candidate outputs with required fragments and forbidden fragments. The plugin does not call the model, does not persist data, and does not claim semantic correctness.

It is released as a DSH bundle. dsh.bundle.patch in package.json points to ./cordis.patch.yml, and cordis.patch.yml automatically registers the tool after the profile’s base tools run.

Core Features

Fragment-level Checks

The plugin supports two types of checks:

  • required fragments: fragments that must appear in the candidate output
  • forbidden fragments: fragments that must not appear in the candidate output

This allows failures to be pinpointed to specific fragments rather than just providing an unverifiable overall judgment.

Deterministic Reporting

The evaluation results include:

  • total passed and failed cases
  • 0..1 score
  • per-case missing or forbidden fragments

This type of output is suitable for CI logs and serves as evidence for threshold-based judgments.

CLI

The plugin comes with a CLI:

  • reads JSON suite
  • prints report to stdout
  • exits 0 if all pass
  • exits 1 if any fail
  • exits 2 for input or usage errors

Exit codes allow CI to succeed or fail directly based on the result.

Installation and Usage

Add the plugin to a specified profile:

dsh plugin --profile <profile> add github:aryswisnu/dsh-eval-regression

The plugin is released as a DSH bundle, and the registration action occurs after the profile’s base tools run.

If you want to enable it locally or from source, follow the steps below:

  1. Clone the repository.
git clone https://github.com/aryswisnu/dsh-eval-regression.git
  1. Enter the directory and install dependencies.
cd dsh-eval-regression
npm install
  1. Build the artifacts.
npm run build
  1. Add to DSH profile from the current directory.
dsh plugin --profile <profile> add .

Typical Usage

Prepare JSON Suite

The CLI reads a JSON suite. Below is an example where includes corresponds to required fragments and excludes corresponds to forbidden fragments:

{
  "suite": "release-smoke",
  "cases": [
    {
      "id": "grounded-answer",
      "actual": "The result is 42. Source: benchmark.csv",
      "includes": ["42", "Source:"],
      "excludes": ["I cannot verify"]
    }
  ]
}

Run Evaluation

Run in an environment where the CLI can be executed:

npx dsh-eval-regression suites/release-smoke.json

If you are already in the plugin repository, you can also use:

npm run evaluate -- suites/release-smoke.json

After running, the CLI prints the report to stdout. The report contains total passed and failed cases, a 0..1 score, and per-case missing or forbidden fragments.

Local Development Checks

The repository provides the following common commands:

npm test
npm run typecheck

These commands are used to verify plugin code locally.

Use Cases and Notes

This plugin is suitable for the following scenarios:

  • Need to check if DSH candidate outputs contain required fragments
  • Need to check if candidate outputs contain forbidden fragments
  • Need to retain repeatable pass/fail evidence in CI
  • Need to manage JSON suites as maintainable files

When using it, pay attention to its boundaries. It is intentionally kept as a narrow deterministic evaluator and does not replace model quality reviews, factual grounding, tool execution checks, or snapshot replay.

The plugin runs with the permissions of the current dsh process; you should check the source code and license before installing. The license has been verified as MIT.

Dependencies and integration information are as follows:

{
  "peerDependencies": {
    "@deepseek-ai/cordis": "^4.0.1",
    "@deepseek-ai/dsh-tools": "^0.1.0-rc.6"
  }
}

dsh.bundle.patch in package.json points to ./cordis.patch.yml.

Links

Directory page:

https://www.skillhub.cn/plugins/aryswisnu/dsh-eval-regression

GitHub:

https://github.com/aryswisnu/dsh-eval-regression