Introduction¶

Under DSH’s “everything is a plugin” development model, the most common pitfalls for plugin authors are rarely found during the compile phase: the build succeeds, unit tests pass, but after release, things might fail due to missing files in the tarball, bundle registration failure in the host, or profile corruption during uninstallation. The commonality of these issues is that they only appear between “the packaged artifact actually installed by the user” and “the real host,” which static checks on the author’s machine cannot cover.

dsh-testkit fills this gap: it places the artifact produced by npm pack into a precisely specified real DSH host, executes the entire lifecycle, and leaves evidence for maintainers to review.

What is it¶

iiwish/dsh-testkit describes itself as “The real-host release gate for DeepSeek Harness plugins,” which serves as the real-host release threshold for DeepSeek Harness plugins. It clearly defines its boundaries: it is a release gate, not a unit testing framework, static linter, model output evaluator, or security authentication. The entire process does not make model calls, does not require model API keys, and is licensed under MIT.

One isolated run answers three release questions:

Lifecycle: From resolve to cleanup¶

A complete isolated lifecycle is:

resolve -> install-dsh -> package -> install-plugin -> assemble -> boot -> register
        -> exercise -> update? -> uninstall -> reboot -> recover? -> cleanup

Each isolated lifecycle tests only one plugin under test; multi-plugin ownership and update order are composition-level issues, outside the scope of this tool.

The adapter currently only accepts specific versions of @deepseek-ai/dsh: 0.1.1-rc.2 (default), 0.1.0-rc.8, 0.1.0-rc.7, 0.1.0-rc.6. Unknown versions terminate with exit code 4 before runner creation; host version drift will not be mislabeled as a plugin failure. The official dsh-v0.1.2-alpha.1 remains a pending canary as the corresponding npm package is unavailable; 0.1.2-alpha.2 only enters a one-time canary matrix. Neither alpha is in the default support matrix.

What Passing Means¶

• The same packaged artifact identified in the report completes all required stages;
• Config lines come from DSH --dump-config, services & tool schema come from in-process Cordis probes;
• Declared exercise runs via the real tool runtime, not through model selection;
• After uninstall, the same profile restarts without the tested bundle, capabilities, or attributable residue;
• Required observers are available; missing required coverage is marked unsupported and does not synthesize a “pass.”

Conversely, passing does not imply arbitrary executable code is safe, model output is good, or unasserted behavior is usable.

Behavior and Cleanliness Verification¶

Means for behavior include: deterministic runtime probes, declared tool calls, optional loopback HTTP route assertions, and explicit browser smoke tests. HTTP and browser traffic are limited to the runner’s own 127.0.0.1; missing Chromium is marked unsupported.

Cleanliness verification for uninstallation covers: same profile restart, capability checks, owned-path residue, process and port checks.

Installation and Enabling¶

Runtime requirements: Node.js 22 or newer, and Docker (Docker is the default runner).

Install as a devDependency:

pnpm add -D dsh-testkit

After installation, the CLI entry is dsh-test. The shortest path is to generate the scenario first, then execute the test:

pnpm dsh-test init
pnpm dsh-test

Typical Usage¶

dsh-test init Generates Scenario¶

dsh-test init runs offline, locates the nearest Git worktree, and generates three reviewable files:

1. <plugin-root>/dsh-testkit.yaml: Contains the exact DSH version and detected line expectations;
2. <repository-root>/.github/workflows/dsh-lifecycle.yml: Default read-only token contract and correct nested paths;
3. <repository-root>/.agents/skills/dsh-testkit/SKILL.md: Enables compatible coding agents to execute the same gate.

Generation is byte-level idempotent and performs pre-checks on all targets; it stops all writes if conflicts are found unless --force is explicitly passed. If the plugin bundle is under the repository root:

pnpm dsh-test init plugin/
pnpm dsh-test --config plugin/dsh-testkit.yaml

Scenario as Code¶

The scenario is YAML with schemaVersion: 1. The initial scenario generated by init looks roughly like this:

schemaVersion: 1
name: my-plugin-quick
subject:
  source: .
dsh:
  version: 0.1.1-rc.2
expect:
  boot: success
  rows: [tool-my-plugin]
  services: [myService]
  tools: [my_tool]
exercise:
  - tool: my_tool
    arguments:
      value: smoke
observers:
  filesystem: required
  process: preferred
  ports: preferred
  network: off
  canary: preferred

expect declares boot results, config lines, services, and tools, while exercise declares tool calls to execute. The local subject directory is mounted read-only and copied to a writable root directory owned by the runner before packing; when prepare, prepack, or postpack are declared, dependencies are restored inside the copy via packageManager and lockfile before npm pack, leaving the original checkout unmodified. The scenario also supports http.routes, update targets, expected failure and recovery, stage rerun, observer strategies, and global watchdogs; see Scenario Reference in the repo for details.

For plugins providing DSH web routes, set profile: web and add Docker-only assertions, for example, requiring /health to return 200:

profile: web
http:
  routes:
    - id: health
      path: /health
      expect:
        status: 200
        json:
          status: ok
          version: $subject.packageVersion

CI Integration¶

The generated workflow uses a read-only token by default and writes this contract into a file:

permissions:
  contents: read

steps:
  - uses: iiwish/dsh-testkit/.github/actions/dsh-test@v0
    with:
      plugin: .
      dsh-version: 0.1.1-rc.2
      config: dsh-testkit.yaml
      publish-junit-check: 'false'

The default behavior writes JUnit annotations to the job, uploads the full evidence directory, and provides an artifact ID, URL, digest, report path, and stable exit code, without calling the Checks API. Trusted push or release workflows can enable named JUnit Check:

permissions:
  contents: read
  checks: write

steps:
  - uses: iiwish/dsh-testkit/.github/actions/dsh-test@v0
    with:
      plugin: .
      dsh-version: 0.1.1-rc.2
      publish-junit-check: 'true'

Enabling publish-junit-check requires the checks: write permission and should not be enabled for untrusted fork PRs. GitHub Enterprise Server and other CI systems can call the CLI directly.

Reports and Exit Codes¶

Reports fall under .dsh-testkit/runs/: the canonical report.json, junit.xml for CI, readable report.md, sanitized command logs, and bounded stage evidence.

The exit codes are stable, allowing scripts to branch directly based on them:

Applicable Scenarios and Notes¶

Who is it for: Plugin authors maintaining DSH plugins, maintainers reviewing release PRs, teams operating plugin templates, and those needing reproducible host-level bug reports.

A few notes before use:

• Each isolated lifecycle tests only one plugin under test; multi-plugin ownership and update order are composition-level issues;
• The adapter only accepts the four specific DSH versions listed above; other versions terminate with exit code 4;
• A “pass” does not imply arbitrary executable code is safe, model output is good, or unasserted behavior is usable, nor is it a security authentication;
• The two alpha versions only enter the canary matrix and are not in the default support scope.

Regarding security: In the DSH ecosystem, plugins run with the permissions of the current dsh process. Source code and licenses should be checked before installing any plugin. dsh-testkit itself is installed as an npm devDependency, is MIT licensed, and source code is public; it does not make model calls, and the CI side defaults to minimum permissions (read-only token).

Summary¶

dsh-testkit pushes release verification from “it runs locally” to “this packaged artifact installs, runs, and uninstalls cleanly on a precise real host, leaving evidence.” For teams maintaining DSH plugins, hanging it on release PRs and tags is a cost-effective way to act as a gatekeeper.

• Plugin Directory Page: https://www.skillhub.cn/plugins/iiwish/dsh-testkit
• GitHub Repository: https://github.com/iiwish/dsh-testkit

The directory page is a community-maintained independent site and has no official affiliation with DeepSeek / Synthflow.