Foreword

When comparing multiple models or Agents in DSH, a specific source of bias is “knowing who the contestants are beforehand”. Once the model identity is seen in advance, the subsequent evaluation of prompts, outputs, and final judgments may be influenced.

dsh-blind-arena addresses this with a local plugin: same task, same commit, isolated worktrees, shared verification, blind evaluation first, then reveal.

What is this

dsh-blind-arena is a DSH plugin released by changer-changer with an MIT license.

It turns the DSH Web into a local Agent arena: same task, same commit, isolated worktrees, shared verification, participant identity is not included in the public experiment record until reveal.

Core Capabilities

  • 2–4 lane races: Real-time progress per lane, including tool calls, turns, and tokens.
  • Anonymity contract: The public wire type ArenaExperiment does not contain an identity field; redactExperiment removes identity-related fields before serving.
  • Shared verification: A user-defined verify command runs for each lane, displaying exit code, timeout, and duration.
  • Blind verdict → reveal: Select the result first, then view per-lane diffs and final answers.
  • Offline export: Export self-contained HTML reports, raw JSON, and per-lane unified patches without network dependency.
  • Winner branch: Promotes the winning lane’s worktree to arena/<id>/<label>, applicable only to real lanes.
  • Zero-cost demo mode: Uses deterministic scripted lanes (including focused, verbose, broken, timeout) to run the full pipeline without an API key.
  • Cleanup: Clean up stale worktrees and experiment states from the UI.
  • Full client UI: Injected into DSH Web, covering create / list / live race / review / reveal.

Installation and Enablement

The runtime environment requires:

Node ^22.19.0 || >=24.0.0
DSH (DeepSeek Harness) with Web client runtime
@deepseek-ai/dsh-agent
@deepseek-ai/dsh-llm
@deepseek-ai/dsh-session

The versions of @deepseek-ai/dsh-agent, @deepseek-ai/dsh-llm, and @deepseek-ai/dsh-session are 0.1.0-rc.7.

Installation command:

npm install dsh-blind-arena

After installation, open DSH Web, and the Arena plugin row will appear in the interface.

Typical Usage

  1. Create experiment: Point to local repo, write task, add verify command, select contestants.

  2. Observe race: View the live progress feed for each lane.

  3. Blind verdict -> Reveal: Judge based on behavior, diff, final answer, tokens, and verification time, then execute reveal.

  4. Export report: Obtain self-contained HTML report, raw JSON, and per-lane unified patch.

  5. When credentials are unavailable, run demo mode first: It uses scripted lanes to go through the full process, facilitating evaluation of the workflow before spending tokens.

Applicable Scenarios and Notes

Suitable for local, repeatable Agent comparisons in DSH Web, especially scenarios where one wants to reduce model identity bias, retain offline reports, and have control over the experiment lifecycle locally.

Notes:

  • The plugin runs with the current DSH process permissions; the source code and license should be checked before installation.
  • The local storage path is:
~/.dsh/arena/
  • Public experiment records do not contain participant identity; identity is stored in the server-side secrets.json and merged into the view only after reveal.
  • Everything is done locally; aside from the browser itself accessing resources, experiment data does not leave the local machine.

Links

  • Directory page: https://www.skillhub.cn/plugins/changer-changer/dsh-blind-arena
  • GitHub: https://github.com/changer-changer/dsh-blind-arena