Introduction

Anyone who has done performance tuning has likely encountered this situation: run a benchmark once, output 1200ms; run it again at a different time, and it becomes 1900ms. Change a line of code to verify if it’s faster, run it once and feel it’s faster, run it again and feel it’s an illusion. Single results carry environmental noise, so directly drawing conclusions from them is unreliable.

DSH’s philosophy is “Everything is a plugin,” and this type of problem also has a corresponding plugin to solve it. Below, we introduce dsh-batch-regression maintained by PangYiMing: run the same command N times, take the median and distribution, and draw conclusions using statistics rather than a single result.

What is it

dsh-batch-regression is a DeepSeek Harness (DSH) plugin. Its positioning in one sentence is: run the same command N times, output the sample count, median, and minimum/maximum values to help you extract the reliable number from the fluctuating data.

It solves three types of problems:

  • Performance data fluctuation: take a credible median;
  • Comparing duration distributions before/after changes;
  • Analyzing reproduction rates for occasional issues.

The license is MIT.

Core Features

The capabilities of the plugin can be broken down into four points:

  1. Configurable rounds. Specify how many rounds to run via ROUNDS, and output the sample count, median, and minimum/maximum values.
  2. Two measurement modes. METRIC=time measures duration and outputs median/min/max; METRIC=success counts success rates or reproduction rates and outputs results in the format PASS=9/10.
  3. Failures in the middle do not interrupt the process. If a round fails, it records a FAIL and continues running the remaining rounds; it will not stop the entire process due to a single round’s failure.
  4. Conclusions are disciplined. The plugin author has distilled several empirical methodologies in the README: run at least 5 rounds (statistics are meaningless with fewer than 5 rounds), take the median instead of the average (the average is skewed by extreme values), and only draw conclusions if the difference before/after exceeds 20%.

Installation and Usage

Install from GitHub by executing:

dsh plugin --profile demo add github:PangYiMing/dsh-batch-regression

The README also provides an npm installation method, but notes that it is available “after publishing to npm”:

dsh plugin --profile demo add dsh-batch-regression

Typical Usage

First, let’s look at an example measuring duration. The command below runs node bench.js 5 times, outputting the sample count, median, and minimum/maximum values:

ROUNDS=5 METRIC=time ./scripts/runner.sh "node bench.js"
# samples=5  median=1234ms  (min=1102ms max=1987ms)

Next, let’s look at an example for counting reproduction rates. The command below runs npm run build 10 times and counts the number of successes, making it suitable for troubleshooting occasional failures:

ROUNDS=10 METRIC=success ./scripts/runner.sh "npm run build"
# PASS=9/10

Execution Discipline and Known Limitations

The plugin README lists several empirical methodologies; it is recommended to read through them before using:

  1. At least 5 rounds. Statistics are meaningless with fewer than 5 rounds.
  2. Take the median, not the average. The average is skewed by extreme values, while the median is more stable.
  3. Conclusions are only stable if the gap > 20%. When comparing before/after, draw conclusions only if the gap exceeds 20%; otherwise, it is better to run more rounds.
  4. Control variables. Use the same machine and time period, close large processes like browsers/IDEs before running.
  5. Acknowledge the sources of fluctuation. Apple Silicon’s dynamic scheduling of P/E cores can double single-run duration, and laptops can thermally throttle after running continuously for ten minutes; these all affect single-run duration.

One empirical lesson worth mentioning separately: the author tried CPU-limiting tools like cpulimit/nice/taskpolicy to eliminate jitter, but empirically found them to fail completely on Apple Silicon. Therefore, this plugin’s approach to jitter elimination is “multi-round median + control variables,” not relying on limiting CPU.

Suitable Scenarios and Notes

Three situations where this plugin is suitable:

  • Performance data fluctuation, wanting to extract a credible median;
  • Comparing duration distributions before/after changes;
  • Analyzing reproduction rates for occasional issues (using METRIC=success).

Unsuitable situations should also be clarified:

  • For matters that can be determined by a single run, just run it once;
  • Locating “which commit introduced regression” is the job of git bisect; the author has another plugin called dsh-bisect-debug;
  • For UI visual regression, use dsh-screenshot-diff.

Additionally, a reminder: DSH plugins run with the permissions of the current dsh process. It is recommended to check the plugin source code and license before installing. The source code for dsh-batch-regression is on GitHub with an MIT license, so you can review it yourself before installing.

Conclusion

When performance data fluctuates wildly, instead of repeatedly running single tests and picking a number based on feeling, let the plugin run enough rounds, take the median, and then look at the distribution. dsh-batch-regression solidifies this practice into two commands. Combined with disciplines like “at least 5 rounds” and “draw conclusions only if the gap > 20%,” the conclusions will be much more stable.

  • Community Directory Page: https://www.skillhub.cn/plugins/PangYiMing/dsh-batch-regression
  • GitHub Repository: https://github.com/PangYiMing/dsh-batch-regression