Introduction

The core design philosophy of DeepSeek Harness (DSH) is “everything is a plugin”. When developing or iterating DSH plugins, how to validate changes without affecting existing workflows is a practical pain point. Releasing directly can easily introduce unexpected regressions. dsh-plugin-abtest provides a solution for paired experiments in isolated environments, helping teams evaluate the impact of changes on task success rate, token consumption, latency, and tool error rate before release.

What It Is

dsh-plugin-abtest is a DeepSeek Harness plugin for reproducible paired A/B experiments before DSH plugin changes. It provides isolated runtime environments, auditable evidence, and deterministic release gates.

  • Maintainer: Morriaty-The-Murderer
  • Category: Workflow
  • License: MIT

Core Features

  1. Paired A/B experiments: Control (current plugin) and Candidate (changed plugin) run in paired fashion in the same environment with the same tasks.
  2. Isolated runtime environment: Each experiment branch has its own DSH_HOME, configuration, and plugin build artifacts, without interfering with one another.
  3. Auditable evidence: Records raw sessions, assertion results, pair-level differences, and reports, supporting Markdown and HTML formats.
  4. Deterministic release gates: Based on rules configured in the manifest, automatically generates one of four decision outcomes: PROMOTE, REVIEW, REJECT, or INCONCLUSIVE.
  5. Reproducible real-world cases: Includes a built-in Toolshrink context budget case, demonstrating how to reproduce an experiment by pinning the DSH version, model parameters, and plugin commit.
  6. Manifest-based configuration: All parameters such as thresholds, repetition counts, and concurrency are managed through the manifest.

Installation and Enabling

This plugin requires running under Node.js ^22.19.0 || >=24.0.0 and pnpm 11.19.0.

Install it to a specified DSH profile (e.g., web) using the following commands:

# 方式一：使用 Git commit SHA（推荐，确保稳定性）
dsh plugin --profile web add 'github:Morriaty-The-Murderer/dsh-plugin-abtest#<full-commit-sha>'

# 方式二：使用 npm 版本号（需确认已发布的版本）
dsh plugin --profile web add dsh-plugin-abtest@0.1.0

After installation, restart the corresponding profile to apply the configuration.

Typical Usage

The plugin is currently an MVP and must be run from source. It mainly contains five steps:

# 1. 初始化实验配置
node --import tsx src/cli/bin.ts init --output ./my-experiment --json

# 2. 冻结当前状态（生成 Control 和 Candidate 的 Artifact）
node --import tsx src/cli/bin.ts freeze --manifest ./my-experiment/experiment.yml --output ./evidence --json

# 3. 运行配对实验
node --import tsx src/cli/bin.ts run --manifest ./my-experiment/experiment.yml --output ./evidence --json

# 4. 生成决策（基于规则判断）
node --import tsx src/cli/bin.ts decision --manifest ./my-experiment/experiment.yml --output ./evidence --json

# 5. 生成报告
node --import tsx src/cli/bin.ts report --manifest ./my-experiment/experiment.yml --output ./evidence --json

After the experiment, detailed evidence files are generated under <output>/<experiment-id>/, where report.html provides a static report view.

Interpreting Results

At the end of the experiment, one of four deterministic outcomes is produced, with the following meanings:

Outcome Meaning Next Steps
PROMOTE Evidence is sufficient, quality meets the standard, and gate rules pass Proceed to manual release process
REVIEW Results improved, but manual judgment is needed regarding cost, latency, or error rate Review pair-level evidence
REJECT Strict gate failed, critical use cases regressed, or benefit too small Fix Candidate and rerun
INCONCLUSIVE Too few valid pairs, or environment not comparable Add evidence rather than treat as failure

The default primary metric is task success rate, and gates can also be set for token usage, P95 latency, and tool error rate.

Applicable Scenarios and Notes

  • Applicable to: DSH plugin developers who need to validate the impact of changes on existing tasks.
  • Runtime: The current version is an MVP and must be run from source (requires Node.js and pnpm).
  • Permissions and security: The plugin runs with the permissions of the current DSH process. Review the source code and license (MIT) before installation.
  • Important note: This plugin only generates offline suggestions (such as PROMOTE) and does not automatically modify real DSH profiles or publish plugins. It is a local validation tool.

Conclusion

dsh-plugin-abtest solves the validation challenge before DSH plugin changes through isolated environments and deterministic rules, helping teams iterate without disrupting existing operations.