Introduction

Writing tests for Agent workflows usually requires handling two types of instability: model outputs and external tools. The official dsh-llm-replay can already replay model streams; however, tools like HTTP, MCP, and databases may still access the network, modify data, or rely on a service that happens to be offline during testing. Additionally, third-party interfaces occasionally return 429 in CI.

dsh-tool-cassette is a community plugin for DeepSeek Harness (hereinafter referred to as DSH). Its approach is straightforward: in the first run, let the selected tools execute realistically and record the normalized results into a cassette file; afterwards, switch to replay mode where real tool body calls are zero, and the recorded results are fed back into the tool chain based on exact matching. Compared to hand-written mocks, it records the DSH-normalized values/errors, and during replay, the output contract must be re-verified. Below is an introduction to its capabilities, installation methods, and usage boundaries.

What is this

DSH’s philosophy is “everything is a plugin,” and dsh-tool-cassette is a community plugin under this mechanism. Its positioning is written in the package description: a plugin for deterministic recording, integrity checking, and offline replay of the DeepSeek Harness leaf tool boundary.

A few basic points need to be clarified first:

  • Independently developed and maintained by community member Lem0nTea2002. It is an unofficial community plugin with no affiliation to DeepSeek official, and has not been audited or endorsed by the official team;
  • License is MIT;
  • Current version 0.1.0, fixed compatibility with DeepSeek Harness 0.1.0-rc.8 (this version is currently on the npm next tag). The same rc.8 CLI must be used continuously for installation and execution;
  • Node engine requirement ^22.19.0 || >=24.0.0; peerDependencies are @deepseek-ai/cordis 4.0.1, @deepseek-ai/dsh-tools 0.1.0-rc.8, @deepseek-ai/schemastery 3.18.1.

How Recording and Replay Work

Record: Run once realistically

In Record mode, the tools selected by include (HTTP, MCP, database, local programs) will execute realistically. The plugin writes DSH-normalized values/errors to a versioned NDJSON cassette, containing additionalContexts and cancellation results. Unselected tools execute according to the original flow; the recorder does not interfere.

The recording process is fixed:

  1. Create <file>.partial exclusively;
  2. Append header, call/start, call/result frames sequentially;
  3. Sync each frame to disk;
  4. After all calls are complete, write a complete tail frame;
  5. Close the file, atomically publish the formal file in create-only mode, then delete the partial.

The artifact comes with an integrity protocol: each frame contains a continuous seq, the hash of the previous frame, and the SHA-256 of the current frame; the last frame is complete. Truncation, duplicate frames, hash tampering, protocol version errors, and missing tail frames will all be blocked before Replay is activated. The recorder refuses to start if the formal file or partial already exists; it preserves .partial and refuses to publish the formal file if there are unfinished calls.

Note that the hash chain does not include digital signatures: it can detect transmission corruption, truncation, and general tampering, but against an attacker capable of rewriting all frames and hashes, the artifact library should supplement signatures, WORM, or immutable storage.

Replay: Four identity exact matching

During Replay, the plugin determines if it is the same call based on four identities:

  1. The structural path of the tool in the call tree;
  2. The tool name;
  3. Lossless JSON parameters after recursively sorting object keys;
  4. The ordinal assigned based on the start order of the call.

Matching is exact: the order of object keys in the parameters object does not matter; { "city": "武汉", "unit": "c" } changing to { "unit": "c", "city": "武汉" } will still hit; however, if the array order, parameter values, call order, or structural path changes, the replayer immediately returns CASSETTE_MISMATCH. Concurrent calls are assigned ordinals based on the start order, supporting reverse-order completion; calls that start earlier and finish later will not be mismatched.

After a Replay hit, real tool body calls are zero. The saved successful values are re-verified against the current output schema, renderer, presentation meta, and post-strategies. Therefore, you still need to register tools with the same name during replay and maintain output contract compatibility: the cassette is only responsible for saving results, while the security checks are still performed by the current DSH execution.

Failure and Closure on Mismatch

The first trajectory deviation puts the replayer in a poisoned state, causing subsequent selected tools to continuously fail, with real body execution skipped, avoiding a hybrid situation where half reads the cassette and half touches the real service. The other situations also have clear handling:

  • Parameters, path, order, or tool name mismatch: Return CASSETTE_MISMATCH;
  • Replay has extra calls or unconsumed records: Headless/CI process exit code is 1;
  • Higher-priority plugins short-circuit tools/execute and tools/post-execute: Record is invalid, Replay is poisoned;
  • Selected call is rejected in pre-execute or guard phase: Fail and close based on the trajectory that did not enter the cassette.

Error diagnostics only show the tool, path, ordinal, and parameter fingerprint; they do not echo back raw parameters, results, raw lines, or absolute paths.

Configuration and CLI

Configuration has only three items

interface Config {
  mode: 'record' | 'replay'
  file: string
  include: string[]
}
  • mode: record executes real tools and writes the artifact; replay hits exactly and skips the body;
  • file: Path to the formal cassette file; during recording, use the same path with a .partial suffix;
  • include: Explicitly select leaf tools, supporting exact names and * wildcards; must not be empty; empty ranges, empty paths, and duplicate patterns fail directly when the plugin starts.

Verify the tape, then play it

dsh-tool-cassette verify .dsh-cassettes/weather.tool-cassette.jsonl
dsh-tool-cassette inspect .dsh-cassettes/weather.tool-cassette.jsonl

verify fully validates the protocol, frame pairing, continuous ordinals, hash chain, and completion tail frame, expressing results via exit codes; inspect displays protocol version, number of tools, number of calls, and consumption instructions. Validation failures only output structural reasons and do not echo raw lines, tool bodies, or absolute paths.

Installation and Activation

Prerequisites: pnpm is installed, and the Node version satisfies ^22.19.0 || >=24.0.0. The plugin is fixed compatible with DSH 0.1.0-rc.8; use the same rc.8 CLI for the installation and running profile.

Install command:

pnpm dlx --package @deepseek-ai/dsh@0.1.0-rc.8 dsh plugin --profile headless add dsh-tool-cassette

The installation package injects a default disabled tool-cassette entry via cordis.patch.yml. First, do the recording: override it in the profile’s cordis.patch.yml, filling in mode, file, and leaf tool scope.

- id: tool-cassette
  name: dsh-tool-cassette
  disabled: false
  config:
    mode: record
    file: .dsh-cassettes/weather.tool-cassette.jsonl
    include:
      - weather_lookup
      - mcp_*_read

After running a real call once, change mode to replay, keeping the rest of the trajectory consistent:

config:
  mode: replay
  file: .dsh-cassettes/weather.tool-cassette.jsonl
  include:
    - weather_lookup
    - mcp_*_read

After the steps above, when calling weather_lookup, the cassette will yield the recorded result, and the real tool body will no longer execute.

If you want to build from source and install the local package:

pnpm install
pnpm run demo
pnpm pack
pnpm dlx --package @deepseek-ai/dsh@0.1.0-rc.8 dsh plugin --profile headless add .\dsh-tool-cassette-0.1.0.tgz

Relative paths are resolved based on the DSH process working directory.

A Demo of Offline Replay

The built-in demo (pnpm run demo) can run the full process:

  1. Start a pure local HTTP weather tool;
  2. Record once, the tool body and network request each occur 1 time;
  3. Close the HTTP service;
  4. Replay the same call, both tool body and network requests become 0;
  5. The return result remains consistent, and all records in the cassette are consumed.

The entire demo does not call the model and does not generate paid API requests.

Additionally, model stream replay is handled by the official dsh-llm-replay; this plugin only manages the leaf tool boundary. Combined, they can build keyless tests where both model streams and tool streams can be replayed.

Applicable Scenarios and Boundaries

It is suitable for these situations:

  • Doing offline regression testing for Agent workflows;
  • Reproducing an expensive or intermittent tool response in CI;
  • Verifying that after plugin upgrades, the current schema, renderer, and post-strategies can still handle old results;
  • Debugging HTTP, MCP, database, or local program tools without repeatedly touching the real service.

The scope of V1 capabilities is: single Agent, single scenario; explicitly selected leaf tools; success, structured failures, additionalContexts; concurrent start order and reverse-order completion; pre-call cancellation without consuming records. The following are explicitly not supported or left for future versions:

  • Multi-Agent and concurrent subagents;
  • Selecting composite tools and their child tools simultaneously;
  • concludesTurn: true;
  • Fuzzy matching, parameter ignoring, automatic fixture updates;
  • Timing simulation for delays and stream cancellations;
  • UI, cloud artifact libraries, benchmark DSL, model judges.

There are two behavioral points to know: after a replay hit, the corresponding record is consumed, and subsequent cancellation in the post-phase will not rollback the consumption position; the replay consumption state only exists in the current process; when all are consumed, it unloads successfully; poison, extra calls, or unconsumed records will cause the headless/CI process exit code to become 1. It does not act as a cache, model replayer, or production idempotency layer.

Security and Usage Notes

To ensure exact replay, V1 saves the normalized parameters, successful values, failure information, rendered content, and attached contexts as-is. Cassettes should be treated as key files or test database snapshots:

  • Default .gitignore already excludes cassettes and partials;
  • Only record in isolated local or CI working directories;
  • Manually check all content before sharing;
  • Close real credentials that are no longer needed after recording ends;
  • Parameter fingerprints have no salt; low-entropy parameters may still be guessed.

V1 does not provide automatic desensitization, encryption, signing, or remote artifact libraries.

Finally, a general reminder: as a community plugin, dsh-tool-cassette runs with the permissions of the current dsh process. Before installing, you should check the source code and license (MIT) and confirm the compatible DSH version before integrating.

Conclusion

The problem dsh-tool-cassette solves is specific: turning a single real tool execution into a verifiable, reusable offline test artifact, preserving sensitivity to trajectory drift through exact matching and failure closure. It guards one side with the official dsh-llm-replay, and together they cover offline regression for model streams and tool streams.

  • GitHub: https://github.com/Lem0nTea2002/dsh-tool-cassette
  • Community Plugin Directory Entry: https://www.skillhub.cn/plugins/Lem0nTea2002/dsh-tool-cassette (The directory is an independent site with no official affiliation to DeepSeek or Huaquan).