Introduction¶

When adding web scraping capabilities to DSH agents, two common problems are encountered. The first is the different page types: SPA and JS-heavy pages must go through real browser rendering to get content, while ordinary article pages use extraction APIs faster, often requiring maintaining two sets of solutions. The second is that DSH’s ctx.web.registerSearchProvider throws WEB_PROVIDER_AMBIGUOUS when registering multiple providers for the same capability, leaving the model with no choice.

dsh-web-fetch (author runfali, MIT license, category: Internet tools) handles both problems simultaneously: wrapping CDP browser rendering and Tavily Extract into two independent LLM tools, allowing the model to choose autonomously based on context. Below is an introduction to its design, installation, and usage.

What is it¶

One-sentence positioning: A dual-source web content scraping plugin for DeepSeek Harness. Each data source is registered as an independent DSH tool, each with its own description and schema. The LLM selects based on context, rather than relying on hardcoded rules.

The division of labor for the two tools:

Key Features¶

Independent Dual-Source Registration, Bypassing Multi-Provider Limits¶

Each data source is an independent DSH tool, allowing the model to choose autonomously. Both tools can be enabled/disabled independently, and disabled tools are completely hidden from the LLM.

Zero Intrusion, Zero Extra Dependencies¶

This is a Cordis bundle plugin, requiring no modification to the DSH core. It is entirely implemented via ctx.tools.register and settings.installSection (requires dsh >= 0.1.2-alpha). Dependencies are only three packages: @deepseek-ai/dsh-settings, @deepseek-ai/dsh-tools, and @deepseek-ai/schemastery.

Config Hot Reload¶

The settings interface is a UI card with hot reload for ~/.dsh/settings.yaml. Changing configuration does not require a restart. lib/client.js includes built-in Chinese (zh) and English (en) interfaces, implemented with React.

Concurrency Safety and Pluggable Strategies¶

The tool declares isConcurrencySafe: true and supports AbortSignal. Adding a new data source only requires implementing the FetchStrategy contract (1 file), and registering it with one line in src/index.js.

Installation and Activation¶

Runtime requirements: Node.js >= 22, DSH >= 0.1.0-rc.7. First execute pnpm install in the plugin directory, then register the plugin, and finally restart DSH:

cd /data/dsh-workspace/dsh-web-fetch
pnpm install

dsh plugin --profile web add /data/dsh-workspace/dsh-web-fetch
# Available after release:
# dsh plugin --profile web add dsh-web-fetch

# systemd environment:
sudo systemctl restart dsh

Skipping the pnpm install step is not optional: The Cordis loader only resolves dependencies from the plugin directory; skipping it will result in ERR_MODULE_NOT_FOUND: @deepseek-ai/schemastery.

Configuration¶

Open Settings → Plugin Config → General Web Content Fetching (web-fetch). At the top are the CDP / Tavily enable checkboxes, below are two groups of configuration items:

• CDP group: CDP Endpoint (default http://10.200.0.5:9222), Timeout ms (60000), Extra wait after load ms (2000)
• Tavily group: Endpoint (https://api.tavily.com/extract), API Key (leave empty to disable), Timeout ms (30000)

After saving, write to the web-fetch: section of ~/.dsh/settings.yaml and hot reload. You can also override using a profile by editing ~/.dsh/profiles/web/cordis.patch.yml:

- id: web-fetch
  config:
    cdpEnabled: true
    cdpEndpoint: 'http://10.200.0.5:9222'
    cdpTimeoutMs: 60000
    cdpWaitMs: 2000
    tavilyEnabled: false
    tavilyEndpoint: 'https://api.tavily.com/extract'
    tavilyApiKey: ''
    tavilyTimeoutMs: 30000

Note: Overriding config replaces the entire configuration block and must include all keys.

Typical Usage¶

After restarting, the LLM will automatically see these two tools, requiring no extra instructions. Manual testing:

User: Use web_fetch_tavily to extract the body of https://example.com
User: Use web_fetch_cdp to scrape https://example.com, a page that requires rendering

The tool output structure is as follows:

{
  "sources": [{ "url": "...", "title": "...", "snippet": "...", "content": "...", "provider": "cdp|tavily" }],
  "truncated": false
}

Adding a New Data Source¶

Implement the FetchStrategy contract in src/strategies/my.js:

export function makeMyStrategy(config) {
  return {
    id: "my",
    title: "My Fetcher",
    available() { return Boolean(config.apiKey) }, // Lightweight check, no I/O
    async fetch(req, signal) {
      return { sources: [{ url, title, snippet, content, provider: "my" }], truncated: false }
    },
  }
}

Then register it with one line in src/index.js:

import { makeMyStrategy } from "./strategies/my.js"
ctx.tools.register(makeToolDef("my", makeMyStrategy, "myEnabled", current))

If you need new configuration items, optionally add fields in Config and FIELD_VIEWS in lib/client.js. No need to modify routing or core logic.

Testing¶

The repository comes with offline unit tests that do not call a real browser or Tavily:

node tests/test-cdp-unit.mjs   # 21 tests
node tests/test-tavily-unit.mjs # 9 tests

Use Cases and Considerations¶

Suitable for two types of people: DSH agent developers who need to cover both rendering pages and lightweight extraction; and maintainers who want to extend DSH via plugins without touching core code.

Considerations before use:

1. The plugin runs with the permissions of the current dsh process; it is recommended to check the repository source code and license (MIT) before installation.
2. tavilyApiKey is stored in plaintext in ~/.dsh/settings.yaml (settings system, not a credential vault). In sensitive environments, it is recommended to override it via the profile’s cordis.patch.yml.
3. The CDP implementation uses Node’s native http with manually written WebSocket frames (no ws dependency), does not enable permessage-deflate, and is compatible with cloakbrowser’s default configuration (compression disabled).
4. Version compatibility follows @deepseek-ai/dsh-settings / dsh-tools / schemastery.

Following the steps above, DSH now has a set of pluggable dual-source scraping tools: the model autonomously chooses rendering or extraction, and adding a new source only requires one file and one line of registration. Repository and Directory page:

• GitHub: https://github.com/runfali/dsh-web-fetch
• Community Plugin Directory Entry: https://www.skillhub.cn/plugins/runfali/dsh-web-fetch