Introduction

DeepSeek Harness (DSH) is an agent development tool that requires metrics feedback during execution. Existing monitoring solutions may face data leakage risks or require complex configuration. As part of the DSH plugin ecosystem, dsh-prometheus provides an out-of-the-box monitoring solution.

Plugin Introduction

This plugin is maintained by xxiaoxiong, categorized as admin-security, and licensed under the MIT license. It aims to provide privacy-friendly Prometheus metrics and Grafana dashboards for DeepSeek Harness.

Monitoring Scope

The plugin covers multiple dimensions of DSH runtime:

  • Sessions and Agents: Monitors active sessions, agent turns, steps, errors, and durations.
  • LLM Layer: Monitors request rates, terminal states, latency, and token counts for input/output/reasoning/cache.
  • Tool Calls: Monitors tool call rates, normalized states (success/error), and latency.
  • Approvals and Subagents: Monitors approval requests, subagent launches, results, and durations.
  • Background Tasks: Monitors background task states (active/launched/completed/killed/failed) and durations.
  • System Diagnostics: Monitors process start times and cardinality overflow diagnostics.

Installation and Configuration

First, install the plugin into DSH’s monitoring configuration set and verify that the configuration layer is inserted correctly:

dsh plugin --profile monitoring add dsh-prometheus
dsh --profile monitoring --dump-config

After installation, the plugin defaults to auto mode. This mode makes the following decisions upon activation:
1. If DSH’s webServer is already active and securely bound, register the /metrics endpoint on the existing routes.
2. Otherwise, start a standalone monitoring endpoint on 127.0.0.1:9464.

If you need to force use of the standalone endpoint (for example, in a Docker environment), you can configure it in cordis.patch.yml:

- id: prometheus
  config:
    enabled: true
    mode: standalone
    host: 127.0.0.1
    port: 9464
    path: /metrics
    allowRemote: false
    maxLabelValues: 64
    maxLabelValueLength: 80

Usage Examples

  1. Start the local stack: Use the provided Docker example to start Prometheus and Grafana.
    docker compose -f examples/docker-compose.yml up -d
  1. Scrape metrics: Verify the endpoint is available with curl.
    curl http://127.0.0.1:9464/metrics
  1. PromQL query: View agent throughput.
    sum(rate(dsh_agent_turns_total[5m]))

Privacy and Limitations

  • Data privacy: The collector never exports prompts, user/assistant messages, system prompts, tool parameters, or results. Only four runtime-controlled label keys are exported: provider, model, tool, and job kind.
  • Remote exposure: Remote exposure does not include built-in authentication or TLS; read SECURITY.md before enabling 0.0.0.0 or remote WebServer routes.
  • Cardinality limits: Label values are constrained by maxLabelValues (default 64) and maxLabelValueLength (default 80); values that exceed these limits or are invalid are grouped as __other__.
  • Upgrades and removal: Treat each DSH upgrade as a compatibility event; removing the plugin does not delete stored Prometheus data.

Notes

  • Version compatibility: The current version is the 0.1.0 developer preview; compatibility testing is only for DSH 0.1.0-rc.6.
  • Dependency requirements: Requires Node.js ^22.19.0 || >=24.0.0 and Cordis ^4.0.1.
  • Permissions and source: The plugin runs with the permissions of the current DSH process; review the source code and license before installation.

Ecosystem Context

The DeepSeek Harness philosophy is “everything is a plugin.” The plugin is available in the community catalog https://www.skillhub.cn/plugins/xxiaoxiong/dsh-prometheus, and has no official affiliation with DeepSeek / High-Flyer.