dsh-plugin-compare

You have several Agent presets configured but cannot say which one finishes faster, burns fewer tokens, or retries less. dsh-plugin-compare pulls execution evidence from session logs and lays it out as a synchronized side-by-side timeline with measurable paired deltas and a statistical summary. It can replay existing sessions to surface write and edit diffs, or launch a controlled A/B run: the workspace is copied twice, the same provider and model are pinned, an identical prompt is submitted, an optional success-check command is executed, and a Git snapshot is captured before the temporary copies are removed. Reports export as JSON, self-contained HTML, SVG, or PNG.

Controlled runs support 1 to 10 paired trials with automatic alternation of order. The summary reports mean and median paired deltas for time and tokens, and a Student-t 95% confidence interval is included when at least two pairs exist. Raw observations and Session ids remain in the JSON export so anyone can recompute the numbers. If an Agent fails before completing its task, the trial is labeled invalid rather than silently folding a no-change test failure into the preset score.

This is for developers running prompt comparison experiments, engineering teams that need a reviewable artifact behind a claim that one preset outperforms another, and anyone who wants to turn subjective feel into auditable, shareable data. Before sharing, review every exported HTML, JSON, SVG, and screenshot carefully: prompts, source diffs, command output, and Session ids may be present, and the best-effort redaction pass is not a privacy clearance.