Preface

When writing DeepSeek Harness (DSH) plugins, many regression issues are not immediately exposed in local unit tests: whether the tool schema passes the real registry, whether the event sequence is complete, whether exception injection causes a crash, whether concurrent re-entry is stable, and whether unexpected footprints are left after mounting. These issues usually need to be verified within the DSH pipeline.

dsh-windtunnel is prepared for this scenario: it replaces “model decisions” with a script but retains the real DSH pipeline, allowing plugin authors to run contract regression in a deterministic environment with zero API keys and zero network access.

What is it

dsh-windtunnel is a DSH plugin, repository owner is BotonJ, current version is 0.1.0, license is MIT.

Its one-sentence positioning is: a contract regression testing chamber for DSH plugin authors.

It mainly solves three categories of problems:

  1. Plugin contract stability: contract issues like loading, registering, validating, rendering, and cancellation.
  2. Whether behavior is assertable: tool calls, result returns, and session event sequences conform to expectations.
  3. Whether regression can be integrated into CI: independent of real models, independent of networks, and results are deterministic.

Its core approach is “Script Adapter Driving the Real Pipeline”: the calls seen by the plugin can come from a script, but the DSH pipeline the plugin enters is actually running. This allows testing the plugin’s contract and behavior within the pipeline without consuming real API keys.

What it can do

The following introduces verified testing capabilities.

Layered Testing

dsh-windtunnel provides four layers of testing:

  • L0 Loading
  • L1 Contract
  • L2 Behavior
  • L3 Injection

Specific Checkpoints

Based on verified functionality, it supports the following checks:

  • Non-ctx footprint snapshot diffing
  • Session event sequence assertions
  • Concurrent re-entry testing
  • Negative case expectFail support
  • Isolated subprocess execution and crash isolation
  • Two usage modes: CLI and bundle

Boundary Statement

Its boundaries need to be clarified here:

dsh-windtunnel is a contract regression net, not a utility testbed.

It verifies “whether the pipeline produces the expected event stream and results when a tool is called with given parameters,” but it does not prove that a real model will actively and correctly call the tool.

Installation & Usage

Install to a specific profile:

dsh plugin --profile <name> add github:BotonJ/dsh-windtunnel

There are two usage methods after installation.

CLI Mode

CLI mode is more suitable for CI. It runs cases directly and outputs a report:

node bin/windtunnel.mjs cases --timeout 90000 --md report.md

Exit code meanings:

  • 0: All passed
  • 1: Failures exist

Bundle Mode

Bundle mode is used to trigger within a DSH conversation. After the plugin is installed into the profile, say to DSH:

帮我跑一下插件风洞

The model will call the windtunnel_run tool to execute the test.

Typical Usage

Writing a Case

Case files are placed under cases/, for example:

cases/xxx.case.mjs

The case object needs to contain the following fields:

name
profile
disableRows
tools
sutPatch
script
expect

Where:

  • name: Case name
  • profile: The profile used
  • disableRows: Adapter rows that need to be disabled
  • tools: Tools of interest for this test
  • sutPatch: Patch configuration for the plugin under test
  • script: Script input
  • expect: Expected assertions

Regarding path resolution, when paths are involved in a case, resolve them relative to import.meta.url; do not hardcode absolute paths.

Using Negative Cases

If the expectation for a case is “a contract violation should be caught,” you can add:

expectFail: true

At this point, the result semantics are reversed:

  • A red result means the contract violation or assertion failure was correctly caught, and the case passes.
  • A green result, on the contrary, means the wind tunnel did not detect the expected failure, and the case fails.

Negative cases are suitable for checking issues like: declaring a timeout but ignoring cancellation signals, incomplete handling of malformed input, incomplete rendering results, missing event sequences, etc.

Execution Model

The execution model of dsh-windtunnel emphasizes isolation:

  • The plugin under test runs in an isolated subprocess.
  • The wind tunnel host is responsible for orchestration and assertions.
  • When the plugin under test crashes, the subprocess exits, but the wind tunnel can still collect results and output a report.
  • Real network adapters are disabled inside the subprocess, configured via overlay disabled: true to prevent network access.

This means the test environment is closer to an isolated state where “physical network access is impossible,” making it suitable for deterministic regression.

Applicable Scenarios

It is suitable for the following users:

  • Authors writing DSH plugins
  • Developers needing to maintain plugin contracts
  • Teams wanting to integrate plugin regression testing into CI
  • Plugin maintainers wanting to check event sequences, tool contracts, injection failures, and concurrent re-entrance

Precautions

Pay attention to permissions and security boundaries before use.

Permissions

The plugin under test runs with full permissions in the subprocess.

Therefore, before putting unreviewed source code into the wind tunnel, static security checks should be completed first, then wind tunnel testing. The verified requirement is: pass the sentinel static security check first, then enter the wind tunnel.

Pre-installation Checks

Before installation, you should check:

  • Source code
  • License
  • Dependencies
  • Plugin entry file
  • Paths of the plugin under test referenced in the case

Known Limitations

dsh-windtunnel has the following known limitations:

  • Utility gap: It does not prove that a real model will actively and correctly call the tool.
  • Dual-line remounting is a proxy testing method.
  • Footprint snapshots are best-effort.
  • DSH breaking changes during the rc period might affect the wind tunnel itself first.

Development and Self-Testing

When developing locally, you can use the following commands:

node --test test/engine.test.mjs

Run dog food cases:

node bin/windtunnel.mjs cases

Running cases requires DSH CLI to be installed on the local machine.

Conclusion

The value of dsh-windtunnel lies in turning DSH plugin contract issues into repeatable, assertable checks that can be integrated into CI. It does not replace real model utility testing, but provides plugin authors with a clear regression net: which contracts are broken, which events are missing, which injections will crash, and which negative cases should have failed can all be run locally.

GitHub repository:

  • https://github.com/BotonJ/dsh-windtunnel