Web Scraping Assistant

Problem

Web scraping often stalls after the URL is collected: field definitions are unclear, selectors are unstable, pagination duplicates results, null or abnormal values are hard to audit, and handoff lacks acceptance criteria. This skill helps market research, operations, product, and data teams turn public page collection goals into executable and verifiable deliverables, not just general advice.

How It Works

It structures output into understanding confirmation, usable results, rationale, acceptance checklist, and next steps. Core capabilities include:

• Collection planning: start from URL scope and a field table for product pages, list pages, or detail pages.
• Page parsing: define selection rules and fallbacks for titles, prices, parameters, and other fields.
• Pagination and deduplication: handle complete list capture with a pagination strategy and dedupe keys.
• Quality checks: inspect nulls, duplicates, anomalies, and missing fields.
• Change monitoring: track page changes by scrape frequency and changed fields.

The workflow usually begins by confirming that data is publicly accessible and reasonably usable, then limits domains, page types, frequency, and fields. It validates parsing rules on small samples, then adds deduplication, retries, rate limiting, and failure logs, and finally compares samples against the original pages with collection timestamps.

Boundaries

It does not bypass login, CAPTCHAs, paywalls, robots restrictions, or access controls, and it does not collect personal privacy data, sensitive identity information, or non-public data. When a request crosses boundaries, it should identify what cannot be done directly and provide compliant alternatives, required materials, or human handoff points. It fits scenarios with clear goals and source material where reviewers need checkable scripts and acceptance criteria, not requests to bypass protected platforms.