Introduction

When performing DSH profile or patch adjustments, the common problem is not “can it run once”, but whether it can repeatedly compare multiple combinations, and put quality, Token, cost, latency, and stability into the same auditable result. young-tim/dsh-profile-lab is a local experiment and release gate tool for DeepSeek Harness, licensed under MIT. Below are its core capabilities, installation commands, typical usage, and boundaries to pay attention to.

What is it

DSH Profile Lab is a DSH plugin maintained by young-tim. It repeatedly runs multiple DSH profile/patch combinations in an isolated workspace, calculates quality, Token, cost, latency, and stability based on official persistent session events, and outputs auditable reports.

Core Capabilities

Its capabilities include:

  • Repeatedly running multiple DSH profile/patch combinations in an isolated workspace.
  • Calculating quality, Token, cost, latency, and stability based on official persistent session events.
  • Outputting report.json, report.md, report.html auditable reports.
  • Providing three DSH tools: profile_lab_run, profile_lab_compare, profile_lab_gate.
  • Providing a “Profile Combination Comparison” tab in the Web profile session.
  • Providing the profile-lab-author-cases Skill to assist in planning test cases.
  • Supporting schema --check, run, compare, gate CLI commands.
  • Supporting release gate policies min_candidate_pass_rate, max_pass_rate_drop_pp, max_median_token_increase_pct, max_error_rate.
  • Result directory contains manifest.json, journal.json, run-state.json, and supports recoverable execution.

Runtime Environment

Verified dependency requirements from the source materials are as follows:

  • Node.js: ^22.19.0 || >=24.0.0
  • pnpm: 11
  • @deepseek-ai/dsh: 0.1.0-rc.7

Installation and Enablement

Verified installation commands from the source materials are as follows:

dsh plugin --profile headless add github:young-tim/dsh-profile-lab

After installation, three DSH tools profile_lab_run, profile_lab_compare, profile_lab_gate become available in DSH. The “Profile Combination Comparison” tab is also provided in the Web profile session.

Typical Usage

Validate, then Run, then Compare and Gate

Real runs may call models and incur costs. It is recommended to first perform schema --check to validate the configuration, then run to generate results, then compare to view differences, and finally gate to determine if release gate requirements are met.

node dist/cli.js schema --check examples/experiment.yml
node dist/cli.js run examples/experiment.yml --output .profile-lab/real-run
node dist/cli.js compare .profile-lab/real-run
node dist/cli.js gate .profile-lab/real-run --policy examples/policy-pass.yml

Following the steps above, you will obtain comparison results and perform release gate judgments based on the policy file.

Let AI Assist in Designing Cases

Verified example prompts from the source materials are as follows:

Help me design a set of Profile Lab cases for the payment retry feature. First check existing experiments and cases, confirm the test scope, key risks, assertions, and budget with me, then give case matrix suggestions; do not create or run files immediately.

The focus of this example is to confirm scope, assertions, and budget before creating or running files.

Run All Cases or Smoke Cases

Example of running a full experiment:

Run all cases in experiments/payment/experiment.yml, save results to a new .profile-lab/payment-2026-08-20. Tell me the number of schemes, cases, repetitions, and estimated total model calls before starting, and wait for my confirmation.

Example of running only smoke-tagged cases:

Use experiments/payment/experiment.yml to run only cases with the smoke tag, output to a new .profile-lab/payment-smoke-001. Do not reuse old result directories.

Executing Release Gates

Verified gate example from the source materials is as follows:

Execute release gate on .profile-lab/payment-2026-08-20: candidate pass rate at least 95%, drop relative to baseline not exceeding 2 percentage points, error rate not exceeding 1%, and median Token growth not exceeding 15%. Explain the reason for each failure.

This example covers the gate policy items of candidate pass rate, pass rate drop relative to baseline, error rate, and median Token growth.

Results and Security Boundaries

The result directory will contain the following files:

  • manifest.json
  • journal.json
  • run-state.json
  • report.json
  • report.md
  • report.html

The materials state that by default, current DSH login credentials are temporarily reused, and cleaned up immediately after the subprocess ends; credentials will not enter manifest.json, journal.json, or the report.

Environment variables are not inherited by default, only PATH, HOME, isolated DSH_HOME, and explicitly allowed list names are preserved. The runtime environment will reject special files such as symlinks, sockets, devices, and FIFOs in the workspace.

Use Cases and Notes

DSH Profile Lab is suitable for DSH developers who need to compare DSH profile/patch combinations, audit Token, cost, latency, and stability, and need to judge release gates based on policies.

Points to note:

  1. Real runs may call models and incur costs.
  2. Third-party DSH plugins run with the current user’s permissions; source code should be checked before installation.
  3. License is MIT.
  4. Cloud hosting, account systems, and automatic plugin installation are not within the scope of this product.

Links

  • GitHub: https://github.com/young-tim/dsh-profile-lab
  • Directory page: The currently verified materials do not provide a directory page URL.