Introduction¶
Agent harnesses (scheduling loops) are prone to failure modes that can invalidate the loop when handling tool calls, state management, and error recovery. Traditional auditing relies on humans reading code, which is inefficient and easy to miss; relying on large models for automatic auditing can produce “hallucinations,” i.e., conclusions not grounded in code.
dsh-harness-audit aims to solve these two problems. It is a workflow-style DSH plugin maintained by Leeaoyin. It performs automated auditing of harnesses using machine-enforced evidence validation. It rejects any finding for which no evidence can be found in the code and generates summary reports based on code rather than generated by models.
Core Capabilities¶
The plugin provides the following core features:
- Fifteen checks: cover every key stage of the agent loop, with each check mapped to a specific failure mode.
- Machine-enforced evidence validation: every finding must be proven by concrete code lines, text, or paths; otherwise it is rejected.
- Self-contained installation: installing the plugin installs all judgment criteria, requiring no additional configuration.
- Code-driven summaries: the final summary in the report is written by code, not generated by a large model, ensuring objectivity.
- Background job execution: audit commands return immediately and run as a background task, without blocking the session.
- Reports and metrics: generates Markdown and JSON reports in
outputDir(default.harness-audit/) and provides a “rejection rate” metric for sub-agent quality monitoring.
Checklist¶
The plugin includes 15 checks grouped into five categories. p1 denotes the seven critical checks (marked with ★). The specific content of each check is as follows:
| Category | Check code | Check content |
|---|---|---|
| State stays self-consistent | C1 | Tool-call pairing integrity: ensure all completion paths (error, timeout, cancellation) are recorded, and that failures in parallel batches do not discard other results. |
| C2 | Append-only history: ensure messages cannot be edited after being added. | |
| C3 | Crash and checkpoint semantics: check state visibility and detectability when a process dies between multi-part writes. | |
| Untrusted input is treated as untrusted | C4 | Model-output parsing: handle malformed arguments, truncated streams, or duplicate call IDs instead of trusting them. |
| C5 | Path and sandbox boundaries: parse and verify whether model-supplied paths are inside the workspace root. | |
| C6 | Secrets and environment: check whether environment variables inherited by model-generated commands contain secrets. | |
| Failure is a first-class outcome | C7 | Error classification and retryability: ensure failures carry explicit error-code classifications (retryable / non-retryable / fatal). |
| C8 | Partial success: check whether independent results are merged into a single state that hides or discards successes. | |
| C9 | Idempotency and side-effect safety: check whether writes, commands, or outbound messages inside a retry wrapper can be executed repeatedly. | |
| Boundaries can be closed | C10 | Cancellation propagation: check whether cancellation signals actually reach outbound requests and child processes. |
| C11 | Timeout layering: check the magnitude relationship between the tool-call layer and underlying resources. | |
| C12 | Loop and budget limits: check hard caps on turns, tool calls, wall-clock time, tokens, and delegation depth. | |
| Finite resources are accounted for | C13 | Context management and truncation boundaries: check whether context is managed as the session grows, and whether truncation breaks call/result pairs. |
| C14 | Prompt-prefix determinism: check byte-level consistency of system prompts, tool definitions, and history messages across runs. | |
| The run is observable | C15 | Trace integrity and replay: check whether the run leaves enough event records to reconstruct the process. |
Execution Flow¶
An audit run consists of three phases:
- Reconnaissance: A sub-agent locates key landmarks (such as the agent loop, request assembly, and tool execution) and reports them via
report_landmark. - Fan-out: A one-shot sub-agent is started for each check, and each sub-agent receives only the landmarks relevant to it and the specific check criteria. If required landmarks are missing, the check is not dispatched.
- Summary: The final report is generated purely by code. The model does not write the summary; instead, code converts “suspected” markers into “confirmed” findings or removes claims without evidence.
Usage¶
The plugin provides a command-line interface and supports specifying check dimensions or running all checks.
Basic commands¶
/harness-audit # Ask which dimensions to audit and show a plain-language menu
/harness-audit C1 # Audit a single dimension
/harness-audit C1,C9 # Audit multiple dimensions
/harness-audit p1 # Run the 7 critical checks (★ marked items)
/harness-audit all # Run all 15 checks
Reports and background jobs¶
After a command executes, it returns immediately, and the audit runs as a background task. Once completed, the agent is woken and reports.
View background job output:
job_output <id>
Report files are saved by default under .harness-audit/, named by time and dimension:
report-2026-08-17_095736-C1.md
report-2026-08-17_095736-C1.json
Evidence Validation and Metrics¶
The plugin includes a strict evidence validation mechanism to ensure the reliability of audit conclusions. report_finding rejects findings in the following cases:
1. The check does not belong to the current run.
2. The path is not first-party code (for example, .venv, site-packages).
3. The file is not inside the workspace.
4. The referenced line does not exist.
5. The referenced text is not within ±3 lines around the referenced line.
6. A “suspected” conclusion lacks a “confirmation hint”.
The rejection rate is the core metric. If the rejection rate is too high, it means sub-agents are fabricating claims, and prompts need to be adjusted. Rejecting unparseable input is also a design principle of the plugin, avoiding silent error handling.
Notes¶
- Installation method: No explicit installation command is provided in the verified facts. The plugin is managed as a DSH plugin; please refer to the official catalog or repository for installation instructions.
- Permissions and scope: The plugin runs with the permissions of the current DSH process. Scope enforcement prevents the plugin from auditing third-party libraries (such as vendored code), ensuring the audit target is the project’s own code.
- False-positive control: Running checks when the target landmark cannot be found is the main cause of false positives. The plugin automatically skips checks whose landmarks are not found.
Ecosystem Context¶
DeepSeek Harness (DSH) adopts the “everything is a plugin” philosophy. dsh-harness-audit is a community-contributed workflow plugin intended to help developers build more robust agent systems. To learn more or get the latest version, visit:
* Plugin catalog
* GitHub repository