dsh-skill-eval

In the DeepSeek Harness (DSH) ecosystem, the precision of skill routing is critical for delivering a seamless user experience. However, ensuring that skill descriptions effectively guide Large Language Models (LLMs) to trigger the correct actions can be a challenging task. The dsh-skill-eval plugin addresses this issue by providing an automated evaluation mechanism designed to rigorously measure the reliability of your skill triggers.

At its core, the plugin leverages an LLM judge to simulate the exact routing logic used by DSH. It reconstructs the official skill catalog prompt and decides, for each test query, whether the target skill should be activated. The results are presented in a detailed report featuring metrics like accuracy, recall, precision, and a confusion matrix, offering a reproducible way to verify how well your skill descriptions route queries.

This tool is ideal for DSH plugin developers, model engineers, and teams building advanced conversational systems. Whether you are validating the logic of a newly introduced skill or testing the performance of different judging models, dsh-skill-eval offers a professional perspective to ensure your skill system performs robustly in real-world scenarios.