dsh-eval-regression

Every iteration of agent code can quietly degrade answers that still look acceptable on the surface. dsh-eval-regression gives DeepSeek Harness a lightweight, deterministic regression gate: maintain a small corpus of expected fragments, and get repeatable pass/fail evidence on every replay or smoke test with no subjective judgment required.

The core tool, evaluate_golden_output, compares a candidate output against two sets of fragments: required fragments catch omissions, and forbidden fragments block known bad claims or unsafe fallbacks. It never calls a model and never persists data; it simply produces a deterministic score from 0 to 1. The plugin also ships a small CLI that reads a JSON test suite, prints a report to stdout, and exits with code 0 for all pass, 1 for any failure, or 2 for invalid input, making it straightforward to block a CI pipeline on a failed expectation.

If you are building a CI baseline for response quality in an agent or LLM application, or you want to treat your evaluation corpus as ordinary, versioned source code, this plugin is a low-effort first gate. It is deliberately narrow and does not replace model-quality review, factual grounding checks, or snapshot replay; it is best used as the first cheap, deterministic checkpoint in a broader evaluation harness.