Introduction

After developing a Web application, agents often report “task completed.” However, code-level self-checks may pass, while the actual page behavior (interaction, style, and visuals) may have issues. Unit tests are often insufficient for visual regressions or real user interaction scenarios.

dsh-verify is a quality gate tool for Web applications built by agents. It provides acceptance testing in a real browser and supports three execution modes: CLI, MCP server, and GitHub Action, ensuring that the application behaves as expected in a real browser environment.

Plugin Overview

dsh-verify is a Playwright-based quality-checking plugin designed to bridge the gap between agent self-testing and the real user experience. Its maintainer is 263311487-ux, and it is licensed under the MIT License.

The core value of this plugin lies in treating the “browser” as an independent arbiter. It does not rely on the subjective judgment of an LLM; instead, it performs real clicks, form inputs, style checks, and screenshot comparisons, returning a deterministic PASS or FAIL result.

Core Capabilities

Based on verified facts, this plugin includes the following core features:

  • Real browser acceptance testing: Powered by the Playwright engine, it performs routine human actions in a browser (clicking, filling forms, checking styles).
  • MCP server support: It can integrate with tools such as Claude Code, Cursor, and Copilot, allowing agents to invoke browser verification during a session.
  • GitHub Action integration: It can run in CI workflows, with every Push able to trigger real browser checks.
  • AI-drafted checklists: It provides the gen command, allowing an AI to generate a test specification from a page URL and a prompt and then execute it automatically.
  • Visual regression: It supports screenshot comparison and diff detection to capture visual drift.
  • Multi-browser support: It supports Chromium, Firefox, and WebKit.
  • JSON-based testing specifications: Test logic is written in JSON format, making it easier to reuse and manage with version control.

Installation and Setup

Before using it, you need to install a Node.js environment and the Playwright browsers.

  1. Install the plugin globally
    npm install -g dsh-verify
  1. Install Playwright browsers
    On first use, download the Chromium browser:
    npx playwright install chromium

Usage

The plugin provides three main usage modes: CLI, MCP, and GitHub Action.

CLI

Run a test specification file or a glob pattern in the command line:

npx dsh-verify --spec 'specs/*.json'

MCP Server Integration

Add the MCP server to Claude Code or Cursor so that an agent can invoke it:

claude mcp add dsh-verify -- npx -y -p dsh-verify dsh-verify-mcp

GitHub Action Integration

Use the official Action in a CI configuration file:

- uses: 263311487-ux/dsh-verify/.github/actions/dsh-verify@main
  with:
    spec: demo/fixed.json       # 指定规范文件或 glob
    # url: https://staging.example.com   # 可选：覆盖规范中的 base

Notes

  • Runtime environment: This plugin depends on @modelcontextprotocol-sdk, playwright, and pngjs.
  • Node.js version: Node.js 18 or later is required.
  • Permissions and security: The plugin runs with the permissions of the current dsh process; please review the source code and license before installation.
  • Browser dependencies: After first use or after updating the Playwright version, you may need to run npx playwright install again.

Conclusion

As an independent browser acceptance testing tool, dsh-verify decouples JSON specifications from real browser execution, providing reliable regression testing for applications built by agents. Whether used for automated checks in a CI pipeline or interactive verification during development, it provides objective pass/fail feedback.