Introduction

DeepSeek Harness (DSH) agents often lack intuitive feedback mechanisms and deep debugging capabilities when interacting with web pages. Developers need agents to operate pages, inspect details, and maintain context and conversation like a human.

The dsh-browser-annotate plugin solves this problem. It provides a real Chromium browser environment through an Electron desktop app and integrates access to the Chrome DevTools Protocol (CDP). The plugin and the desktop app communicate over local HTTP loopback (127.0.0.1:7391), giving the agent a complete set of browser manipulation tools. It also allows users to right-click annotate elements and directly inject frontend facts into the conversation.

Core Features

The plugin provides twelve browser manipulation tools, covering the full workflow from basic navigation to complex interaction:

  1. openBrowserPage: Opens a new tab in the browser and returns real Chromium errors if the target is unreachable.
  2. navigatePage: Supports URL navigation, refresh, forward, and back.
  3. readPage: Retrieves an accessibility tree snapshot of the page, with selector-based scope filtering.
  4. screenshotPage: Supports viewport or element screenshots, returned as image attachments.
  5. clickElement / hoverElement / typeInPage: Performs clicking, hovering, and typing based on element references or selectors, and supports Enter submission.
  6. dragElement: Executes dragging via the engine, falling back to Playwright’s dragAndDrop if the engine does not support it.
  7. handleDialog: Handles alert/confirm/prompt dialogs instead of automatically answering them.
  8. runPlaywrightCode: Executes custom Playwright functions in the app’s Chromium via CDP.
  9. cdp: Supports calling any Chrome DevTools Protocol method (e.g., Input, DOM, Runtime, Network, etc.).
  10. annotations: Reads and manages the user-created annotation queue.

In addition, the plugin has the following features:

  • Right-click annotation and inspection: Right-click an element in the browser to quickly add an annotation or enter developer mode to inspect the DOM.
  • Annotation queue mechanism: Supports queuing selected element information (including CSS selector, DOM path, screenshot, computed styles, etc.) and later injecting it into the conversation in one step.
  • Two-pane layout: The DSH Web UI is on the left, and the real Chromium browser is on the right, without interfering with each other.

Installation and Activation

To install this plugin, use the DSH plugin command. This repository is not published as an npm package.

# 安装插件
dsh plugin --profile web add github:iimaguest/dsh-browser-annotate

# 更新到最新提交
dsh plugin --profile web update dsh-browser-annotate

# 卸载插件
dsh plugin --profile web remove dsh-browser-annotate

After installation, DSH loads the plugin on startup. If the desktop app is not running, calls to browser tools by the agent will receive a “desktop browser not connected” message instead of crashing.

Usage Examples

  1. Start the desktop app: Run the following command in the terminal to start the Electron app and pass in the DSH Web UI token URL.
    ./dev.sh 'http://127.0.0.1:3090/?token=…'

For subsequent launches, the URL can be omitted; the app will read the `.dev-url` file.
  1. Operate the browser: The agent can use the twelve tools above to interact with pages.
  2. Manage annotations: Users can right-click to add annotations in the browser, or use the commands provided by the plugin to manage the queue.
    # 列出当前队列中的注释（结果附带截图）
    annotations list

    # 将注释队列注入到对话输入框中
    annotations inject

    # 清除已注入或未处理的注释
    annotations clear

Architecture and Limitations

The plugin and the desktop app have separate responsibilities:

  • Plugin side: Provides HTTP routes and tool definitions on the DSH side, without loading any DSH services.
  • Desktop side: Runs the real Chromium environment and handles page rendering and user interaction.

Known limitations:

  • Tab closing: Closing tabs in the Electron app must be done manually (shortcut ⌘W); the agent tools have not yet implemented automatic tab closing.
  • CDP limitations: Electron does not support the Target.createTarget and Target.createBrowserContext APIs, so new browser contexts cannot be created directly by the agent. New tabs can only be opened through openBrowserPage.
  • Startup security: When launching the macOS DMG installer for the first time, system security restrictions usually require right-clicking to open it or manually allowing it under “Privacy and Security”.

Conclusion

dsh-browser-annotate combines a real browser environment, CDP access, and an annotation queue mechanism, filling the gap in DSH for web interaction and deep debugging. It enables agents to use the browser like an operator while allowing humans to provide frontend facts directly to the agent through intuitive annotations.

GitHub repository