dsh-telemetry

Running a multi-model, multi-plugin Harness stack raises one question over and over: where did the time and the money actually go? dsh-telemetry answers it by bolting a local-first telemetry layer onto every request, model call, tool invocation, and plugin hook in the pipeline. For each attempt it records latency, input and output token counts, retried-call overhead, estimated cost (when a pricing catalog is configured), error classification, and cache-hit statistics. Storage stays entirely on your machine—date-partitioned JSONL on Node 18+ or optional SQLite on Node 22.5+—with zero network egress.

On top of the raw store, a rich CLI lets you filter by time window, group by model, plugin, tool, profile, or day, isolate slow requests or errors, and drill into a trace/span tree where every attempt (including rate-limit fallbacks) is timed independently. A read-only Web UI, bound strictly to 127.0.0.1, puts P95 latency, time-to-first-token, cost estimates, and error breakdowns on a single screen, each metric annotated with its sample count and recomputable from the underlying events. Privacy is handled by omission: prompts, responses, file contents, command arguments, and secrets are never captured by default; sensitive headers are dropped whole, URL credentials are redacted, and tool, plugin, model, and profile names support stable-but-non-reversible hashing.

It is built for developers and teams operating multi-model, multi-plugin DSH Harness stacks who want quantified visibility into latency percentiles, cost allocation, and failure patterns without shipping conversation data to a third party. It is also a practical auditing tool for catching circular tool calls, diagnosing abnormal retry loops, and pinpointing which plugin hooks are most likely to block or error.