Preface

In DSH (DeepSeek Harness), whether a skill can be utilized largely depends on its description: the model decides whether to invoke a specific skill based on the description within the skill directory prompt. If the description is too narrow, matching queries will fail to trigger the skill (under-triggering); if it is too wide, irrelevant queries will also be triggered (over-triggering).

These two types of issues can usually only be identified by manually testing a few queries based on intuition, resulting in results that are neither reproducible nor easily comparable. dsh-skill-eval transforms this into a reproducible evaluation: it uses an LLM judge to fully reproduce the official skill directory prompt, judges whether each test query should trigger the target skill line by line, and then summarizes the metrics.

What is this

dsh-skill-eval is a Skill-trigger evaluation plugin maintained by renjianguojinqianfan, licensed under MIT. The current version is 0.1.0, and it requires Node >= 20.

It solves a specific problem: measuring the reliability with which a skill’s description routes matching queries to that skill, and quantifying the degree of under-triggering and over-triggering numerically.

How it works

The plugin operates in four steps:

  1. Enumerates the skills callable by the model in the current session (ctx.skills.snapshot).
  2. Reproduces the official catalog message word-for-word, including <system-reminder>, <available_skills>, and descriptions that have undergone normalization, truncation, and escaping.
  3. For each test query, instructs the judge model to determine if the target skill should be triggered, forcing it to output a single line of YES/NO.
  4. Compares the results with the expected labels and summarizes the metrics.

The summarized metrics include accuracy, precision, recall, false positive rate, false negative rate, and a confusion matrix.

To ensure the reproduced catalog message does not silently deviate from DSH upgrades, the plugin includes a built-in catalog fidelity fixture, locking the official dsh-tool-skill@0.1.0-rc.6 template. After upgrading DSH, you can refresh the fixture from a local official installation and review the diff:

node scripts/refresh-catalog-fixture.mjs <path-to-dsh-tool-skill/lib/index.js>

Installation and Enablement

The installation command must be executed in the repository root (repo root):

dsh plugin --profile web add ./dsh-skill-eval

Then configure the judge model routing in cordis.patch.yml of the profile/overlay:

- id: skill-eval
  config:
    provider: <provider-id>
    model: <model-name>

Note that the provider here must already be registered in the DSH LLM runtime, which is the same runtime used by your profile chat. The plugin will validate this route upon startup and issue a warning if the provider is not registered.

Writing Test Cases

A test case is a JSON array where each item contains a query and the expected label should_trigger:

[
  { "query": "add a tool to the harness that persists across restarts", "should_trigger": true },
  { "query": "help me write a Python script for this CSV", "should_trigger": false }
]

The category field is optional and is currently reserved for future use.

Typical Usage

Slash command:

/skill-eval <skill-name> [test-file]

Model callable tool:

run_skill_eval(skill_name="<skill-name>", test_file="examples/dsh-plugin-eval.json")

The test-file parameter is optional and defaults to examples/dsh-plugin-eval.json within the plugin package; relative paths are resolved relative to the plugin package directory.

If you want to compare the judgment differences of different judge models, simply change the provider/model in the configuration and run again.

Development and Testing

The plugin comes with a set of development and testing scripts:

npm run check     # All JS file syntax checking
npm test          # node:test, including official catalog fidelity and mock ctx testing
npm run smoke     # 51 pure function smoke assertions
npm pack --dry-run  # Check publish file list
bash scripts/mount-smoke.sh  # Real DSH mount smoke test in a temporary home

Applicable Scenarios and Notes

Suitable for:

  • Developers who write or maintain skills for DSH and want to systematically verify the quality of description routing.
  • Developers who want to discover under-triggering/over-triggering issues before submission rather than relying on luck after going live.

Notes when using:

  1. The evaluation results measure the routing accuracy of the judge model for a given description, not the accuracy of the target skill itself. Results may vary if you switch judge models.
  2. The judge’s provider must be registered in the DSH LLM runtime; otherwise, the plugin will issue a warning upon startup.
  3. The plugin runs with the permissions of the current dsh process. You should check the source code and license (this project is MIT) before installing.

Conclusion

For those writing DSH skills, dsh-skill-eval provides something previously missing: a reproducible, comparable number that tells you exactly how many matching queries are correctly routed to the skill by the description.

DSH’s philosophy is “everything is a plugin,” and the community maintains an independent plugin directory (with no official affiliation to DeepSeek or HF). Project entry points are as follows: