Preface

In the DeepSeek Harness (DSH) plugin ecosystem, agents frequently need to read webpage content and then convert it into summaries, keywords, link structures, or image analysis results. If the model were to assemble HTTP requests, parse HTML, and truncate body text each time, the code would be repetitive and error-prone. dsh-scrape-webpage consolidates this type of work into a single DSH composite plugin: it scrapes http/https webpages, extracts titles, descriptions, body text, H1–H6 structures, and links, outputs statistics and high-frequency keywords, and supports downloading content images and integrating an image analyzer.

What Is This

dsh-scrape-webpage is a Host-only, zero-dependency DSH composite plugin maintained by 131CDA1 under the MIT license. It targets scenarios involving “reading webpage content and performing lightweight analysis.” The plugin is distributed as a bundle, with dsh.bundle.patch declared in package.json. After installation via dsh plugin, it registers as a composite layer.

Core Features

Below are the main capabilities it provides.

  • Web Scraping: Supports http/https with automatic redirects; non-2xx status codes are returned gracefully.
  • Content Extraction: Extracts titles, page descriptions, body text, H1–H6 heading structures, and link lists; link lists are capped at 100.
  • Statistical Analysis: Outputs character count, word/character count, heading count, link count, estimated reading time, language inclination (Chinese/English/Mixed), and top 30 high-frequency keywords.
  • Image Downloading: Can download page content images and return local paths for analysis via read_image.
  • Image Analysis Extension: Publishes the scrape.imageAnalyzer service; when an image analysis plugin registers an analyzer, image analysis results are appended as imageAnalysis in the output.
  • Sandbox Authorization: Scraping is performed within the sandbox by default; if HTTPS is restricted, one-time authorization can be obtained via sandbox_permissions and approval.

Images will be downloaded to a local directory similar to:

.scrape-images\<timestamp>\

Installation and Activation

First, ensure pnpm is in your PATH; dsh plugin forwards to pnpm. On first use, it automatically initializes the target profile.

Install from npm Registry

Here is the recommended npm registry installation command:

dsh plugin --profile web add dsh-scrape-webpage

Install from GitHub

If installing from GitHub, use:

dsh plugin --profile web add "github:131CDA1/dsh-scrape-webpage"

Local Directory Debugging

Local directory debugging requires explicit use of the file: prefix. Bare paths or relative paths will be treated as link: protocol by pnpm, which may cause the package to be unresolvable.

dsh plugin --profile web add "file:D:\path\to\dsh-scrape-webpage"

Uninstallation

Uninstall command:

dsh plugin --profile web remove dsh-scrape-webpage

Typical Usage

After the steps above, the most direct way to use it is to say in a conversation:

Help me scrape https://example.com and analyze it

Common Parameters

Common parameter names are as follows:

url
maxChars
images
sandbox_permissions
justification

Description:

  • url: The http/https address to scrape.
  • maxChars: Maximum number of characters to return for the body text, default 30000, range 1000–100000.
  • images: Number of content images to download simultaneously, capped at 6, default 0.
  • sandbox_permissions: Only supports danger-full-access, used for retry when HTTPS sandbox access is restricted.
  • justification: Used with sandbox_permissions to explain why broader permissions are needed.

Scraping with Images

If images are needed, first let the scraping plugin download them, then read the returned local paths:

Help me scrape https://example.com and analyze it, images: 3

The response will include the local relative path relPath for each image. Then, by calling read_image on relPath, the image can be handed over to a vision model for analysis.

Integration with Image Analysis Plugins

If there is an independent image analysis plugin, you can register an analyzer via the following service:

ctx.get('scrape.imageAnalyzer')

After registering an analyzer, scraping with the images parameter set greater than 0 will pass images to the analyzer and append the imageAnalysis result in the output.

Security and Limitations

  • Protocols: Only supports http/https.
  • Limits: Link list capped at 100; image downloads capped at 6; maxChars defaults to 30000, range 1000–100000.
  • Timeouts: Tool 180s, page 30s, single image 20s.
  • Sandbox: Scraping is performed within the sandbox by default. If HTTPS sandbox TLS access is restricted, the model must explicitly include the following parameters for retry:
sandbox_permissions="danger-full-access"
justification="A one-sentence explanation of why broader permissions are needed"

Approval authorizes only the current invocation.

  • Platform: Deployments without a fetch provider invoke curl.exe via shell; it is included in Windows 10 1803+ and must be in PATH on other platforms.
  • Host Dependencies: tools, systemPrompt, and timer are hard dependencies; web, shell, sandboxPolicy, approval, and sessions are optional and will report graceful errors if missing.
  • Permissions: The plugin runs with the permissions of the current dsh process and may rely on host services to initiate network requests and download images. Before installation, review the source code and MIT license.

Use Cases and Considerations

This plugin is suitable for the following scenarios:

  • When you need to convert webpage body text into analyzable text.
  • When you need statistics such as titles, links, keywords, language inclination, and estimated reading time.
  • When you need to download page content images and integrate with vision or image analysis.

If the task requires non-http/https protocols or exceeds the body text, image count, or timeout limits, the plugin itself does not cover these capabilities and requires additional handling.

Conclusion

The value of dsh-scrape-webpage lies in consolidating webpage reading, structural extraction, statistical analysis, and image analysis integration into a single plugin, reducing repetitive implementation. The project address:

https://github.com/131CDA1/dsh-scrape-webpage