Introduction

In the DeepSeek Harness (DSH) ecosystem, packaging real-world reading experiences into tools is key to improving agent capabilities. dsh-toutiao-reader is exactly such a plugin: it encapsulates the author’s hands-on experience with processing Toutiao articles into the webfetch tool and the toutiao-reader skill.

Static pages on Toutiao are often blocked by its anti-crawler mechanisms (such as __ac_nonce and __ac_signature). Direct requests may return a security verification page, while web_search often returns only a summary. This plugin fills that gap: it takes an article URL and returns the title and the full plain-text content.

Core Features

The plugin mainly consists of two parts:

  1. webfetch tool
    Implements a three-level fetching strategy to ensure a high success rate:

    • Direct connection: Static direct fetch, about 1 second (uses a domain cache to remember which sites require a browser).
    • Resident Edge CDP: Cold start takes about 20 seconds; after startup, it listens on port 9333. Subsequent requests take about 3–6 seconds. It reuses persistent tabs and proactively blocks images and ByteDance CDN to improve speed. It also supports automatic handling of security verification pages (challenge pages).
    • dump-dom fallback: A one-click rendering fallback when CDP is unavailable.

  2. toutiao-reader skill
    Packages a complete set of model invocation documentation, covering:

    • Toutiao URL pattern recognition.
    • Reading the method field to determine the fetch level.
    • Handling challenge pages and controlling retry pacing.
    • Managing the resident headless browser (port/PID/stop command).
    • Troubleshooting short or empty text results (for example, video pages and app interstitial pages).

Installation and Enablement

Command-Line Installation

Use the official CLI command to add the plugin:

dsh plugin --profile web add dsh-toutiao-reader

Plugin Market Installation

In the DSH Web interface:
1. Go to Settings.
2. Find Plugin Market.
3. Search for “toutiao” and install it.

Local Development Installation (Manual)

If you are doing local development, manually create a Junction and modify the configuration:

New-Item -ItemType Junction -Path "$env:USERPROFILE\.dsh\profiles\web\node_modules\dsh-toutiao-reader" -Target "<plugin dir>"

Then modify the profile’s package.json: add "dsh-toutiao-reader": "link:<plugin dir>" to dependencies, and add "dsh-toutiao-reader" to dsh.profile.bundles.

Configuration

In the DSH Web cordis.patch.yml file, enable id-override mode (overwrite only the ID; do not re-insert):

- id: dsh-toutiao-reader
  config:
    workspace: D:/DSH_Workspace  # 相对 outfile 路径的基目录
    cdpPort: 9333                # CDP 调试端口
    browserPath: ""              # 浏览器路径，留空则自动检测 Edge 或 Chrome
    maxChars: 12000              # 默认文本截断长度
    domainCachePath: ""          # 域名缓存路径，留空则使用 %TEMP%/webfetch-domains.json

Common Usage

Calling webfetch

The model calls webfetch(url), and the returned structure is as follows:

{
  "method": "resident-edge-cdp",  // 抓取方式
  "title": "文章标题",
  "text": "文章正文内容...",
  "textLength": 1024,
  "truncated": false,
  "outfile": null,
  "error": null
}

Parameter description:
* url (string, required): The article URL. The PC-side www.toutiao.com/article/<id>/ URL works best; mobile/share pages usually have no article body.
* maxChars (number): The maximum length of the returned text, default 12000, maximum 60000. If exceeded, truncated is set to true.
* outfile (string, optional): Save the full text (title + blank line + body) to a file. Supports absolute paths or paths relative to workspace.

Notes

  1. Cold-start latency: The first invocation of the resident browser takes about 20 seconds; subsequent invocations take about 3–6 seconds. This is normal.
  2. Read-only safety: The plugin operates in read-only mode and does not write local files unless outfile is specified. It does not read any credentials and does not involve private data such as mobile device data.
  3. Port restriction: The resident headless browser listens only on 127.0.0.1 and is not exposed externally.
  4. Environment requirements: Edge or Chrome must be installed locally (auto-detected), and Node.js version must be ≥ 20.
  5. Concurrency restriction: isConcurrencySafe: false. Because CDP reuses a single tab, invocations must be serialized.
  6. Text content: If the returned text is very short or empty, it is usually caused by video pages, Q&A pages, or app interstitials. Use a PC-side URL instead.

Conclusion

dsh-toutiao-reader solves the problem of difficult Toutiao article scraping by combining a three-level strategy: direct connection, resident browser, and dump-dom. It can be used as a standalone plugin or share the resident browser process with workspace scripts. It is suitable for developers who need to process news or article content in batches within the DSH environment.

  • GitHub: https://github.com/RyanShen3/dsh-toutiao-reader
  • Directory: https://www.skillhub.cn/plugins/RyanShen3/dsh-toutiao-reader