Introduction

Technically, running a DSH agent through an existing benchmark is not difficult; the challenge lies in leaving behind verifiable evidence after completion: ensuring that results were produced using fixed protocol, fixed seed, and fixed parameters, verifying the JSON artifact fields, and ensuring the final report does not covertly turn running numbers into trading capabilities (alpha claims). The ml-quant-trading repository defines these issues clearly—deterministic synthetic inputs, fixed protocol v1 commands, JSON artifacts suitable for automated checking, public issue templates, and a clear boundary: benchmark throughput is not equal to trading performance.

To have an agent execute the “Run → Verify → Summarize → Draft Report” chain reliably, each step needs to be registered as a tool. The dsh-plugin-mlquant-benchmark introduced below does exactly that.

What is This

dsh-plugin-mlquant-benchmark is a DeepSeek Harness plugin maintained by initial-d. A one-sentence positioning: A DSH toolset for reproducing the ml-quant-trading protocol v1 benchmark. The license is MIT, and the current version is 0.1.0.

It solves a specific problem: allowing DSH agents to run existing benchmarks, read machine-readable artifacts, validate against the benchmark protocol, and draft a ready-to-submit issue report. The boundaries are equally clear: it does not add a trading agent, does not call market data APIs, and does not configure any model provider.

The README frames the challenge in one sentence: Can DeepSeek Harness reproduce a quantitative benchmark end-to-end, preserve the evidence package, and not turn running numbers into alpha claims? This plugin is the answer to that question.

Core Features

The plugin registers four DSH tools:

  • mlquant_benchmark_v1_cpu: Runs the fixed protocol v1 CPU benchmark, writing results to artifacts/benchmark-v1.json.
  • mlquant_read_benchmark_json: Reads the JSON artifact and renders it into a compact Markdown result table.
  • mlquant_validate_benchmark_json: Validates protocol v1 fields, expected use cases, fixed parameters, and variance warnings.
  • mlquant_draft_github_issue: Drafts the DeepSeek Harness benchmark issue body from the JSON artifact; it will not be published to GitHub.

These four tools correspond to the four steps of the report chain: run first, then read, then validate, and finally draft. The publishing step is left to human action.

Installation and Activation

Execute the installation command in a DeepSeek Harness profile or preset environment:

dsh plugin --profile web add github:initial-d/dsh-plugin-mlquant-benchmark

The package declares a dsh.bundle manifest (patch: ./cordis.patch.yml), with a bundle id of mlquant-benchmark, and inserts the following line:

- id: mlquant-benchmark
  name: dsh-plugin-mlquant-benchmark

Two points to know: this package is deliberately not published to npm yet and is currently distributed solely via GitHub; dependencies are @deepseek-ai/dsh-tools ^0.1.0-rc.7 (appearing in both peerDependencies and devDependencies).

Typical Usage

After the above installation and configuration, the typical usage path is the Run-To-Report Path in the README, consisting of four steps:

  1. Install the plugin from GitHub.
  2. Open an initial-d/ml-quant-trading checkout in DSH.
  3. Let DSH run, validate, summarize, and draft the benchmark report.
  4. Submit the drafted report via a dedicated issue template.

This path is deliberately kept minimal. During specific execution, the DSH prompt suggested in the README is:

Read AGENTS.md, docs/benchmarking.md, and docs/reality_check.md.
Use the mlquant benchmark tools to run the protocol v1 CPU benchmark, validate
and read the JSON artifact, and draft a DeepSeek Harness benchmark report. Keep
the result as an engineering reproducibility benchmark, not a trading-performance
claim.

Note the requirement at the end of the prompt: keep the result as an engineering reproducibility benchmark, not a trading-performance claim.

The public report path uses the dedicated template in the main repository:

https://github.com/initial-d/ml-quant-trading/issues/new?template=deepseek_harness_benchmark.yml

For a seed example, see https://github.com/initial-d/ml-quant-trading/issues/61. To understand the background and the guardrails for agents, you can read the DeepSeek Harness Recipe and Quant Agent Reproducibility Target documents in the main repository.

Local Development

If developing with a local checkout, manually add the same line to the manifest:

- id: mlquant-benchmark
  name: file:/path/to/dsh-plugin-mlquant-benchmark

The development commands are:

npm install
npm test

Tests will load the plugin using a mocked ctx.tools.register, verifying that all four tools are registered, reading and validating example artifacts, and drafting an issue body.

Applicable Scenarios and Notes

Suitable for: DSH developers who want to verify if the agent harness can reproduce a quantitative benchmark end-to-end and fully preserve the evidence package. This plugin does not provide investment advice, does not claim backtest performance, the artifact does not contain private data or API keys, and the tools will not publish anything to GitHub—the submission of reports is always a manual action.

Additional points to note:

  • The plugin runs with the permissions of the current dsh process; the source code and license should be checked before installation (this project is MIT).
  • The plugin has no hidden model provider configuration.
  • It is not part of a trading system, only responsible for the benchmark reproduction chain.

Conclusion

The value of dsh-plugin-mlquant-benchmark lies in fixing a chain that is prone to deviation: directing DSH’s attention to running, validating, reading, and drafting, producing a reproducible engineering benchmark report, rather than an investment or leaderboard claim. The plugin has been listed in awesome-dsh-plugin (via PR #2573).

  • Community Directory: https://www.skillhub.cn/plugins/initial-d/dsh-plugin-mlquant-benchmark
  • GitHub: https://github.com/initial-d/dsh-plugin-mlquant-benchmark