XScrapy AI-Native Generic Web Scraper Engine

Problem

Web pages lack a uniform schema. Product details, news articles, forum posts, and documentation pages expose fields in different DOM structures, so hand-written selectors are brittle and time-consuming. x-scrapy targets the narrow problem of turning a URL into clean, structured JSON with less per-site boilerplate.

How It Works

The workflow is:

• Detect the page type from the target URL and match a scene template such as ecommerce, news, social, forum, docs, or listing.
• Extract fields from the template, for example title, price, in_stock, and specs for product pages, or author, article_body, and published_date for articles.
• Use the CLI, Python API, or single-page extraction mode, with optional proxy, anti-detection, and JS rendering settings.
• When rules miss information, enable AI extraction: analyze page semantics, extract requested fields, fill gaps left by rules, and return results with a confidence_score.

Boundaries

• Best for list pages, detail pages, news pages, and documentation pages where the goal is stable JSON output.
• SPA or AJAX-heavy pages may require JS rendering; otherwise the engine may capture only the initial HTML shell.
• AI extraction incurs external API costs, so it should be used when rule-based extraction is insufficient.
• Results include confidence scores; low-confidence fields should be reviewed. Use it in line with target sites' robots.txt and terms of use.