Smart Scraper Web

Problem

Web fetch tools often return raw HTML, leaving agents to manually locate title, tables, prices, lists, and article text. Page structures vary, and repeated requests to the same page add latency.

How It Works

smart-scraper-web fetches pages using Node.js http/https and parses them with bounded regex patterns into structured output rather than full HTML. Core modes include --extract for title, headings, paragraphs, links, images, tables, lists, prices, and metadata; --table, --list, and --price for targeted extraction; --article for heading-plus-paragraph focus; --parse for already-fetched HTML; and --status for cache inspection. The cache is stored in memory/scraper-cache/cache.json, with a default 5-minute TTL and LRU eviction capped at 50 entries or 10MB. Security controls allow only public http/https URLs, block file://, data:, gopher://, localhost, private IPs, and cloud metadata endpoints, revalidate redirect targets, and limit redirects to 5.

Boundaries and Notes

It is not a DOM parser and does not execute JavaScript, so SPA content and frontend-rendered data are out of scope. Price detection is regex-based, recognizing $, €, £, and ¥ with thousands separators, but it does not infer meaning. Each page has a 15-second fetch timeout and a minimum 100ms interval between requests. After scraping sensitive pages, consider deleting cache.json.