Introduction

In DSH, the main Agent continuously generates code, executes commands, and invokes tools. Relying solely on the model’s own judgment or manual checking for multiple candidate solutions, long-task progress, and whether the final session is credible often lacks directness.

The dsh-llm-verifier introduces a configurable independent verification mechanism: the main Agent is responsible for executing tasks, while the plugin then invokes a judge model specified in the settings to verify based on the current session, candidate solutions, and terminal execution evidence.

Below, we will introduce its capabilities, installation method, basic usage, and important limitations.

What It Is

dsh-llm-verifier is a native DSH plugin, positioned as a configurable DSH-native LLM verifier with a web settings page. The repository owner is Aa728848.

The core problem it solves is: after the main Agent completes a round of tasks, it uses an independent judge model to explicitly determine “which candidate is more reliable, how far the task has progressed, and whether the current session meets acceptance criteria.”

Core Capabilities

First, let’s discuss the capability boundaries. This plugin provides four explicit tools:

  • verifier_compare
  • verifier_select
  • verifier_track
  • verifier_current_session

and supports configurable host-level automatic session acceptance.

Automatic acceptance occurs at the agent/turn-stopping lifecycle boundary and is scheduled by stage:

select → compare → track → current_session

Judgments use the A–T fine-grained scale, and the plugin converts the judgment into a 0–1 score. For pairwise comparisons, it reduces positional bias through two rounds of position swaps.

In terms of scoring, the plugin supports probabilistic expected scoring; when the routing does not support logprobs, it falls back to explicit labels.

In scenarios with multiple candidate solutions, the plugin uses the Probabilistic Pivot Tournament to handle candidate comparisons.

The plugin also supports persistent score caching, retries, and telemetry statistics. Successful score caching and invocation statistics are written to the persistent directory of the current DSH topic:

~/.dsh/sessions/<workspace>/<session-id>/verifier/

After the plugin layer is installed, DSH will automatically recognize the dsh.bundle declaration within the package and complete automatic plugin layer alignment without manually modifying any configuration files.

Installation and Enabling

Here, we use the web profile as an example. Installation command:

dsh plugin --profile web add dsh-llm-verifier

After successful installation, DSH will automatically recognize the dsh.bundle declaration within the package and complete automatic plugin layer alignment.

Start the DSH Web client:

dsh web

or:

dsh --profile web

After entering the interface, open:

Settings → LLM Verifier

Here, you can configure the Provider, Model, Reasoning Effort, maximum concurrency, and caching strategy.

Common management commands:

dsh plugin --profile web update dsh-llm-verifier

Update the plugin.

dsh plugin --profile web remove dsh-llm-verifier

Uninstall the plugin.

dsh plugin --profile web list

View plugins installed in the current profile.

Typical Usage

When explicitly invoked, the plugin exposes four tool names:

verifier_compare
verifier_select
verifier_track
verifier_current_session

These are used for pairwise comparison, multi-candidate selection, progress tracking, and current session verification, respectively. In automatic mode, the plugin schedules these tools by stage at the agent/turn-stopping boundary; after enabling automatic acceptance, the final session verification will participate in the acceptance determination.

If you only need to reuse the core algorithm, you can also import it as a regular npm dependency:

pnpm add dsh-llm-verifier

import {
  extractScore,
  extractProgressScore,
  bradleyTerry,
  pivotRoundPairs,
} from 'dsh-llm-verifier/core'

This import is used to reference core scoring and tournament-related functions.

Applicable Scenarios and Notes

It is suitable for use in the following DSH usage scenarios:

  • Multiple candidate solutions need to be compared or selected;
  • Long tasks have multiple checkpoints, requiring progress tracking;
  • An explicit or automatic final acceptance is needed before the session ends;
  • Switching Provider, Model, Reasoning Effort, maximum concurrency, and caching strategy in the web settings page is required.

Important notes:

  1. The plugin runs with the permissions of the current DSH process. Before installation, you should check the source code and license.

  2. Current verified materials do not clearly state the license information, nor do they provide a directory page URL; you should confirm the license terms in the repository yourself before use.

  3. After disabling the enabled tools, explicit invocation will immediately report an error, and automatic gating will not run.

  4. In case of routing or acceptance exceptions, a fail closed strategy is adopted.

  5. If the judge model does not support image input, the corresponding DSH Adapter will explicitly report an error and block it when images are passed in; the plugin will not silently discard image evidence.

  6. The core algorithm originates from the upstream project llm-as-a-verifier/llm-as-a-verifier.

Conclusion

The value of dsh-llm-verifier lies in extracting “evaluating candidates, tracking progress, and verifying sessions” from the implicit behavior of the main Agent and handing them over to a configurable independent judge model for processing.

GitHub URL:

https://github.com/Aa728848/dsh-llm-verifier

The directory page URL is not provided in the verified materials.