The core design philosophy of DeepSeek Harness (DSH) is “everything is a plugin”. In model inference development or operations, quickly validating model availability, throughput (TPS), and first-token latency is a frequent requirement. As a DSH plugin, masknull/dsh-model-tester adds a testing panel at the bottom of the Settings → Models page and performs batch checks through real streaming calls.

Plugin Positioning

This is a model inference plugin maintained by masknull. It is designed to solve the problem of being unable to quickly verify connectivity and performance after configuring a model. With one-click testing, it provides four metrics: availability, TPS, first-token latency, and total duration.

Core Capabilities

  1. Concurrent streaming tests: Performs real streaming calls (llm.stream) for each model, with 3 concurrent streams and a 30-second timeout per model.
  2. Channel-isolated testing: Supports a “Test This Channel” button to send requests only to models from a specific provider.
  3. Single-line multi-metric display: Test results are displayed on one line: availability / TPS (tok/s) / first token (s) / duration (s).
  4. Failure diagnosis: Failed lines explicitly show the error reason, error code, and HTTP status.
  5. Single-model retesting: Each line has its own “Test/Retest” button.
  6. Smart filtering: Automatically hides providers that are inactive or not configured; if a model ID or name contains vision (for example, modlens vision), automatically registered entries are also hidden.
  7. Model list source: Prioritizes models explicitly configured in settings; otherwise, falls back to the adapter catalog (llm.listModels).

Installation and Enablement

Install the plugin using the official command:

dsh plugin --profile web add github:masknull/dsh-model-tester

After installation, restart the dsh web process, then open Settings → Models. A testing panel will appear at the bottom of the page.

Usage Instructions

Go to the bottom of the model settings page and click the “Test” button to run batch tests.

About TPS: This metric depends on the usage information returned by the upstream provider. If the response arrives all at once and the decode window is less than 50 milliseconds, an effective rate cannot be measured, and — is displayed.

About parameters: The test request intentionally omits maxTokens, allowing the upstream provider to use its own default output quota. This avoids premature truncation caused by manually setting a small quota (response incomplete: max_output_tokens).

Summary

By coordinating between the host side and the browser side, this plugin standardizes and visualizes testing workflows that would otherwise require manually writing scripts or calling APIs. It is suitable for scenarios that frequently require model performance and connectivity validation.