Introduction

Under the plugin-based architecture of DeepSeek Harness (DSH), comparative experiments and ablation studies for agent inference Runtimes often suffer from difficult Runtime switching and experiment configurations that are hard to fix and reproduce. harzva/dsh-research-lab is a standalone DSH plugin package designed to provide management of multiple inference Runtimes, fixed Baseline protocols, and freezing and reproduction of experiment Manifests within the DSH Web interface.

Plugin Positioning

  • Maintainer: Harzva
  • Type: Workflow plugin
  • Core Value: Provides a Manifest-first control plane for managing and fixing experiment parameters, generating ablation matrices, and ensuring reproducibility of experimental results.

Core Features

This plugin provides the following capabilities:

  1. Runtime Management
    Manage multiple inference Runtimes within DSH Web. Supported types include:

    • Implemented: ReAct Default, Plan Guided, Code Orchestration
    • Design state: Plan-and-Execute, Tree of Thoughts (ToT), Reflexion

  2. Baseline Protocol
    Ensure reproducibility of experiments by fixing a set of parameters:

    • DSH version
    • Profile, Preset
    • Model, data, and budget
    • Metrics and seeds

  3. Ablation Matrix
    Generate a combination matrix of Runtime × repetitions, supporting up to 64 planned runs.

  4. Deterministic Manifest
    Output a standardized JSON Manifest and compute a SHA-256 checksum. Combined with private storage in DSH Home, this ensures atomicity and idempotency for experiment records.

Installation and Enablement

Before installing, ensure that Node.js 22+, pnpm 11, and the currently validated DSH version 0.1.0-rc.6 are installed.

  1. Build the plugin package (skip this step if a .tgz file already exists):
    pnpm pack:plugin
  1. Install the generated .tgz file into the DSH Web Profile:
    dsh plugin --profile web add .\artifacts\harness-flow-dsh-research-lab-0.1.0-m0.tgz --save-exact --ignore-scripts
  1. After installation, enable the plugin through the DSH Web interface:
    Path: Settings → Plugins → Research Lab

Typical Usage

After enabling the plugin, you can perform the following operations in the DSH Web interface:

  1. Create a Baseline: Configure a fixed parameter combination in the control plane.
  2. Generate a Matrix: Based on the fixed Baseline, specify different Runtimes and repetitions to generate an ablation matrix.
  3. Freeze the Manifest: Export the experiment configuration as JSON, generate a SHA-256 checksum, and lock the experiment process.

Notes and Boundaries

  • M0 Boundary: The current version is a Manifest-first control plane. It does not directly invoke models, does not incur API costs, and does not claim that Plan-and-Execute, ToT, or Reflexion have full runtime capability.
  • Permissions: The plugin runs with the permissions of the current DSH process. It is recommended to review the source code and license before installation.
  • Community Project: This project is a community project and does not represent official endorsement by DeepSeek.

Summary

harzva/dsh-research-lab provides DSH users with an infrastructure for managing complex inference experiments in the Web interface. By fixing the Baseline and generating a deterministic Manifest, it addresses the pain point of hard-to-reproduce experiment configurations and is suitable for developers and researchers who need to rigorously compare different inference strategies.

  • Project Home: https://github.com/Harzva/dsh-research-lab
  • Plugin Catalog: https://www.skillhub.cn/plugins/Harzva/dsh-research-lab