About this skill

The Problem

Web scraping often fails because pages vary in structure, rely on JavaScript rendering, use anti-bot controls, or sit behind paywalls. Sending raw HTML to an LLM is also noisy and expensive. This skill treats extraction as a staged engineering task, which is useful when you need reliable JSON from articles, metadata, or public entity data.

How It Works

Planning first: identify target URLs, content scope, crawl size, and output format before generating code.
Five-stage pipeline: detect article-like content, fetch over HTTP, parse HTML, render with Playwright when needed, and optionally extract entities with an LLM.
Config-driven selectors: maintain CSS selectors and pipeline settings in YAML, while generated Python scripts emit JSON output.
Credential safety: use os.environ.get() in generated scripts and avoid hard-coded keys, authentication bypasses, or hard paywall circumvention.

Boundaries

It is best suited to public articles, documentation pages, metadata extraction, and entity extraction. It does not promise to bypass login pages, paid content, or aggressive anti-bot systems. Stage 5 requires OPENROUTER_API_KEY; earlier stages generally need no credentials. Frequent site redesigns still require selector and output-quality monitoring.

Screenshots

Use Cases

Batch-extract titles, publish dates, and article text from public pages into unified JSON.
Check whether a site needs JavaScript rendering, then generate rerunnable Playwright scripts.
Maintain CSS selectors in YAML and periodically extract metadata from public documentation pages.
Extract entities from public pages in scripts, returning JSON with an error field on failure.