Web Scraping and Content Understanding

Problem

Web scraping often stalls on how to fetch, clean, and actually use page content. Feeding raw HTML to an LLM wastes tokens and increases entity-extraction noise; regex alone struggles with natural fields such as body text, title, author, and date; bulk crawling also tends to hit anti-bot controls, rate limits, encoding issues, and lost progress.

How It Works

The skill turns the job into auditable Python scripts, YAML configs, and JSON output: it first uses lightweight HTTP requests and page signals to detect article-like pages, then escalates to text cleaning, bulk crawling, Playwright rendering, and optional LLM entity extraction. The LLM only sees cleaned text, not raw HTML; CSS selectors are kept configurable so site changes are easier to maintain.

Boundaries

It fits public web pages, article bodies, metadata, and simple entity extraction; it is not intended to bypass hard paywalls, authenticated pages, or unauthorized logins. It should respect robots.txt, apply rate limiting, and check encoding. If the optional entity-extraction stage is used, OPENROUTER_API_KEY should be set in the runtime environment, not hard-coded into generated scripts.