Web Fetch Toolkit
Paste the following prompt into your AI chat to install this skill:
Please follow https://skillhub.cn/install/skillhub.md to install @user_f28fdad8/oc-web-fetch.
About this skill
Problem
When feeding web content to LLMs or analysis pipelines, raw scraping quickly requires handling encoding, main-content noise, batch deduplication, element extraction, and Markdown conversion. @user_f28fdad8/oc-web-fetch reduces that workflow to four scripts: single-URL fetch, batch fetch, selector extraction, and HTML-to-Markdown conversion.
How It Works
fetch.py issues an HTTP request, detects encoding, extracts the main text, and returns title, body, and metadata. batch_fetch.py fetches multiple URLs through a concurrent queue and reports success/failure counts plus elapsed time. extract.py supports tag, class, id, and attr selectors, returning only matched text or attributes. to_markdown.py removes navigation, ads, scripts, and styles, then preserves headings, paragraphs, lists, links, and images as clean Markdown. The inputs and outputs are standardized as JSON where possible, making it easier for agents or downstream programs to consume.
Boundaries
It is best for static pages and ordinary HTML body extraction. Its text-density-based cleaning does not replace browser rendering, login sessions, JavaScript-heavy UIs, or complex interaction flows. Batch fetching uses conservative default concurrency and delay; in production, check target-site robots, rate limits, and compliance constraints.
Use Cases
- Fetch one competitor page, extract title, body, and metadata, then convert it to Markdown for model analysis
- Fetch multiple source URLs concurrently and produce a summary of success/failure counts plus elapsed time
- Use class or id selectors to extract a target section and store the matched text or attribute list
- Clean legacy HTML pages and convert them to Markdown while preserving headings, lists, links, and images
Best For
- RAG engineers: turn public documentation pages into clean Markdown and JSON
- Data analysts: batch fetch multiple URLs and summarize success, failure, and timing
- Frontend or crawler engineers: extract target regions using class or id selectors
- Data journalists: clean news pages and convert them into editable Markdown
Related Skills
Scrapes JD search results for product titles, prices, sales, and stores, then exports verified data to UTF-8 CSV.
An A-share market monitor using Sina Finance API for real-time quotes, price and percentage alerts, daily logging, and scheduled checks.
Turn pasted or queried Xiaohongshu notes into a research memo covering title patterns, structure, quoted comments, and follow-up angles; public lookups are optional and confirmed before paid execution.
A universal base component that automatically launches the Edge browser and connects the BrowserSkill extension, ensuring readiness for web scraping and browser automation tasks.