Web Content Extraction

Problem

When an engineer pastes a WeChat, Zhihu, API-doc, or skills.sh URL into an agent, the hard part is not opening the link; it is separating clean body content from navigation, lazy-loaded media, anti-scraping shells, and rendered UI. web-content-extraction treats the URL as input and produces Markdown text and media references that can be analyzed, summarized, or translated.

How it works

The skill selects tools by reliability and cost:
- Primary path: web_extract pulls article content into Markdown and fits ordinary web pages, some WeChat articles, and PDFs, handling up to 5 URLs per request.
- Discovery aid: web_search finds candidate pages by keyword or site: operator when no exact URL exists.
- Fallback path: when anti-bot controls, dynamic rendering, or login walls block extraction, browser_extract reads the page through browser_navigate and browser_snapshot.

For WeChat articles, the notes are explicit: repeated web_extract failures may be quota exhaustion rather than a site fault. A more reliable route runs scripts/fetch_weixin.sh to fetch HTML with curl, then parses the js_content region for body text, author, image data-src values, and media lists. If the user provides only keywords, the Sogou WeChat search flow resolves real URLs before full-text extraction.

Limits

This is a text-first extraction capability: body text and image URLs are usually recoverable, while video, audio, and image-only content may depend on JavaScript or lazy loading. Sogou search may rate-limit requests, and real URL resolution can fail occasionally. Content behind login and large PDFs that need splitting are outside scope. Extracted content is intended for the current conversation only and is not written to persistent memory.