DeepSeek Harness (DSH) uses a plugin-based architecture, aimed at extending functionality through the approach of “everything as a plugin.” For developers who need to run LiveBench evaluations locally, they usually have to manually manage the environment and configure parameters in the command line. To address this issue, Vithrive developed the dsh-livebench-panel plugin. It integrates the LiveBench evaluation workflow into DSH Web’s Trajectory view, allowing users to initiate evaluations, monitor progress, and view scores directly in the interface, without leaving the existing DSH ecosystem.

Plugin Positioning

This is a DSH Web plugin that mainly solves the cumbersome configuration problem for local LiveBench evaluations.

  • Name: dsh-livebench-panel
  • Maintainer: Vithrive
  • License: MIT
  • Core functionality: Adds a LiveBench tab to the Trajectory view, supporting configuration of all models, automatic retries, Baseline probing, and real-time result reading.

Installation and Activation

Installing the plugin is straightforward and requires only one command.

dsh plugin --profile web add dsh-livebench-panel

After installation, restart dsh web, open the Trajectory view of any conversation, and you will see the new LiveBench tab on the right side of the “Conversation / Trajectory” tab.

Prerequisite dependencies: This plugin acts only as a console; the actual evaluation is performed by the LiveBench project installed on the local machine. When installing the plugin, the bottom of the panel displays local LiveBench installation instructions. The complete local installation steps (using Windows as an example) are as follows:

:: 1) 克隆 LiveBench 到默认路径，或其它路径
:: 装在其它目录需设置环境变量 DSH_LIVEBENCH_HOME
git clone https://github.com/LiveBench/LiveBench V:\PythonProject\C_UtilizeSpace\LiveBench

:: 2) 使用 Python 3.11 建立虚拟环境并安装（3.10 会报错）
cd /d V:\PythonProject\C_UtilizeSpace\LiveBench
py -3.11 -m venv .venv
.venv\Scripts\python -m pip install -e .

:: 3) 仅评测 coding 类需要安装评分依赖
.venv\Scripts\python -m pip install -r livebench\code_runner\requirements_eval.txt

:: 4) 下载题目数据
cd livebench
..\.venv\Scripts\python download_questions.py

:: 5) 重启 dsh web 并在面板点击刷新

Core Functionality

The plugin provides a complete loop from configuration to result display:

  1. View integration: Adds a LiveBench tab to the Trajectory view, located at the far right of the conversation and trajectory tabs.
  2. Model support: Supports all Providers configured in Harness and their models (including the built-in deepseek-official).
  3. Parameter configuration: Provides dropdowns for configuring various parameters, including reasoning intensity, question set release, category, task, question index range, and max-tokens.
  4. Baseline probing: Provides a Baseline question bank. Users can select a reference model (such as zai-coding-cn__glm-5.3-flash@max) and filter the questions it “got wrong” or “got right.” This allows users to quickly compare the relative strengths and weaknesses of different models against the reference model.
  5. Protocol routing: Supports Anthropic/Claude-family models using the anthropic-messages protocol and OpenAI-family models using protocols such as openai-completions.
  6. Automatic retries: Supports automatic retries for failed requests, mitigating errors caused by API fluctuations.
  7. Score table: Automatically reads the scoring files generated by LiveBench and calculates the average score, without waiting for LiveBench to publish its own leaderboard.

Typical Usage

  1. Initial validation: It is recommended to choose a smaller task (such as typos under the language category) and a cheaper model to quickly verify that the workflow works properly.
  2. Baseline probing: When comparing models, the “wrong” questions of the reference model are selected by default. The value of the probe is to discover difficult questions that the reference model cannot solve, which can effectively distinguish stronger or weaker models.
  3. Post-run operations: After the evaluation task ends, you must restart dsh web and click “Refresh” in the panel to read the latest scores and logs.

Notes

  • Local environment dependencies: You must ensure that LiveBench is installed locally (Git clone, Python 3.11, venv) and that the environment variable DSH_LIVEBENCH_HOME is set to point to that directory.
  • API Key mechanism: The plugin does not directly store the key in the interface. Instead, it parses it from the Harness provider’s apiKeyEnv environment variable and passes it to LiveBench via LIVEBENCH_API_KEY.
  • Special protocol handling: Anthropic/Claude-family models are forced to use the anthropic-messages protocol, and the thinking field is automatically configured based on reasoning intensity (such as medium, high), without sending temperature.
  • Empty answer limitation: If the model’s thinking consumes all max-tokens, resulting in an empty body (an empty result other than $ERROR$), the plugin will not automatically retry. This is a configuration issue; manually adjust max-tokens and rerun.

Summary

dsh-livebench-panel seamlessly integrates LiveBench’s evaluation capabilities into DSH’s Web interface, providing visual configuration, monitoring, and result reading through the Trajectory view. Together with Baseline probing and automatic retries, it significantly lowers the barrier and complexity of local model evaluation.

  • Project homepage: https://github.com/Vithrive/dsh-livebench-panel
  • Plugin directory: https://www.skillhub.cn/plugins/Vithrive/dsh-livebench-panel