dsh-plugin-abtest

Tweaked a prompt, adjusted a threshold, shipped it, and only later found that token consumption doubled or a boundary case quietly regressed. DSH Plugin A/B Test closes exactly that gap: before you release, it runs your before (Control) and after (Candidate) against the same task set in isolated DSH environments and produces quantified, case-by-case evidence so the release decision rests on data rather than gut feeling.\n\nUnder the hood it spawns two independent DSH environments, pairs results case by case, and scores task success rate, token usage, P95 latency, and tool error rate. The final verdict, PROMOTE, REVIEW, REJECT, or INCONCLUSIVE, is determined by deterministic rules you define in the experiment manifest, not by a model deciding on its own. All session events, assertion results, pair-level deltas, and reports are written to disk in JSON, Markdown, and HTML, ready for archival, peer review, or sharing with the team.\n\nIt suits plugin maintainers who want a fast, reproducible quality gate after every change; teams that want to fold plugin releases into CI with a clear pass or fail signal; and anyone who would rather see the numbers say yes than it looks fine. Two modes are available: an offline scripted provider that requires no API key and is ideal for local iteration, and a live-model mode that connects to a pinned DSH version or any OpenAI-compatible Chat Completions endpoint for validation closer to production conditions.