In the DSH plugin ecosystem, handling HTTP requests often requires dealing with anti-bot and rate-limiting challenges. dsh-sxs-anti-bot-http is a dedicated utility plugin for this scenario, designed to encapsulate general anti-bot and rate-limiting strategies and reduce the maintenance cost of web access in agent development.

Plugin Introduction

This plugin was developed by user yangyunsong023. Its strategies are distilled from the SXS collection system, battle-tested with millions of requests per day. It is implemented in pure Node.js and has zero runtime dependencies. The plugin itself is open source and free, licensed under MIT.

Core Features

The plugin mainly provides the following capabilities:

  1. UA pool rotation: Includes 5 browser UA sets, breaks fixed fingerprints, and simulates real browser access.
  2. Exponential backoff retry: Automatically retries on network errors or server 5xx responses, preventing request failures caused by transient instability.
  3. Anti-bot wall detection: Recognizes common anti-bot indicators such as CAPTCHAs, security verification, slider verification, Cloudflare challenges, and Alibaba Cloud WAF.
  4. Site-level adaptive rate limiting: Dynamically adjusts request intervals based on site responses. When consecutive failures occur, intervals back off from 30s→60s→120s→240s→480s; after requests recover, it automatically falls back to prevent IP blocking.

Install and Enable

Use the official command to add the plugin to the DSH configuration:

dsh plugin --profile web add git+https://github.com/yangyunsong023/dsh-sxs-anti-bot-http.git

After installation, the plugin can be used in the web context.

Typical Usage

The plugin provides three core tool functions:

  • sxs_fetch: Anti-bot GET request that returns web page text.
  • sxs_fetch_json: Anti-bot GET request that automatically parses JSON responses.
  • sxs_rate_status: Views adaptive rate-limiting status for each site.

Example code:

// 抓取网页文本
sxs_fetch(url="https://example.com/news", referer="https://example.com/")

// 抓取 JSON 接口
sxs_fetch_json(url="https://api.example.com/data")

Applicable Scenarios and Notes

  • Applicable scenarios: Suitable for DSH agent development that requires frequent access to external websites, especially when target sites have some anti-bot capability.
  • Permissions and security: The plugin runs with the current DSH process permissions. Before installing, developers should review the source code and license to ensure it meets their needs.
  • Dependency notes: The plugin requires GitHub account authentication for installation via the npm package, but installation via the Git link is usually not subject to this restriction.

Summary

dsh-sxs-anti-bot-http provides the DSH plugin ecosystem with a battle-tested anti-bot solution. Through UA rotation, exponential backoff, and adaptive rate limiting, it can effectively improve the stability and success rate of HTTP requests. For more details, see the plugin directory or the project source code.