Introduction

The DeepSeek Harness ecosystem embraces the philosophy of “everything is a plugin.” When developing agents or using DSH, developers often encounter the following problem: the documentation claims that a server supports a certain OpenAI-compatible protocol feature, but actual calls fail due to parameter errors or streaming issues. dsh-compat-probe is specifically designed to solve this problem.

Plugin Introduction

This is a DeepSeek Harness plugin named compat_probe. It sends five micro fixture requests to a target OpenAI-compatible server and returns a “protocol card.” This card consists of a set of boolean values that precisely describe the features the server actually implements, rather than what the documentation claims. It is primarily used to answer questions that can cause crashes at Harness runtime: Is the server online? Does GET /models return a response? Does stream: true return Server-Sent Events? Does the server accept max_tokens, max_completion_tokens, or reject both?

Core Features

  • Protocol compatibility probing: Compatible with the OpenAI server protocol.
  • Fixture testing: Sends five micro requests and returns a protocol card.
  • Implementation detail inspection: Inspects protocol implementation details, including streaming type, max_tokens, max_completion_tokens, and so on.
  • No additional dependencies: Does not include GGUF weights and does not start llama-server or any other process.
  • Offline testing: Supports offline testing (using a node:http mock).

Installation and Enablement

Add it using the official installation command:

dsh plugin --profile web add github:jwilson411/dsh-compat-probe

The installer reads dsh.bundle.patch and appends cordis.patch.yml to the configuration. After installation, ensure that the dependency version is correct; otherwise, output format mismatches may occur:

npm install @deepseek-ai/dsh-tools@0.1.1-rc.2

Configuration

Configure the following fields in the plugin’s cordis.patch.yml:

  • baseURL: The address of the target server, including the API prefix (such as /v1), for example http://127.0.0.1:8080/v1.
  • apiKey: The authentication key. It is sent as Authorization: Bearer; using the environment variable DSH_COMPAT_PROBE_API_KEY is recommended instead of writing it directly into the configuration.
  • allowRemote: Whether to allow probing non-loopback addresses. Defaults to false; it must be explicitly set to true to probe other hosts.
  • model: The model ID used in fixture requests.
  • timeoutMs: The timeout for each request (in milliseconds), defaulting to 3000.

Probe Card

The tool returns an object containing the following fields:

  • reachable: Whether any HTTP response was received.
  • models_ok: Whether GET {baseURL}/models returned a 2xx status code.
  • stream_sse: Whether stream: true returned Server-Sent Events (text/event-stream or data: frames).
  • stream_ignored_json: Whether stream: true was ignored and a non-streaming JSON completion object was returned.
  • honors_max_tokens: Whether the server accepts the max_tokens parameter.
  • honors_max_completion_tokens: Whether the server accepts the max_completion_tokens parameter.
  • http_status: The first non-2xx status code, or the last successful status code, or null (no connection).
  • error_code: The error code (UNREACHABLE, HTTP_ERROR), present only on failure.

Notes

  • Secure defaults: By default, only loopback addresses are probed (localhost, 127.0.0.1, ::1). Non-loopback targets are rejected until explicitly allowed. Loopback determination is based on the address as written, not on its resolved result.
  • Not a testing tool: It is not Ollama, a load tester, or a GPU benchmarking tool.
  • Not a client: It is not a client hosted by OpenAI or Anthropic.
  • No weights/processes: It does not include GGUF weights or download functionality, and it does not start llama-server or any other process.

Conclusion

This tool focuses on protocol verification and compatibility confirmation rather than performance testing. It uses structured data to answer “what the server can actually do,” helping developers avoid runtime errors. For related resources, refer to the directory page and GitHub repository.