JD Search Product Scraper

Problem

JD’s search results are not suitable for a one-pass DOM scrape. The page is an SPA, the product list relies on lazy loading, and items may only render after the viewport reaches them. The page also includes navigation, cart widgets, ad labels, and recommendation blocks. Changing the URL to paginate or assuming first-screen data is complete can produce missing fields, duplicates, or irrelevant items.

How It Works

The skill takes a search keyword $keyword and a page count $pages (default 2) as inputs, then opens the JD search page in a browser. The workflow is:
- Load the search page and verify that the keyword appears in the search box and result area;
- Use evaluate to scroll slowly and wait for lazy-loaded images and prices to render;
- Extract product title, price, sales, and store fields while filtering obvious navigation and cart noise;
- Paginate by clicking the on-page page button rather than constructing a new URL;
- After pagination, compare the first three product titles with the previous page to confirm the list refreshed before continuing;
- Merge data from all pages and export a UTF-8 CSV file, for example jd_{$keyword}_{$pages}pages.csv.

Boundaries and Caveats

This workflow targets JD search pages that are accessible without login. Some limitations remain: sales data may be missing for certain products, leaving the CSV field empty; low-relevance items such as recommendations may still appear if automatic filtering is insufficient; JD page changes can break pagination selectors or extraction rules; login prompts, CAPTCHAs, or risk-control blocks require manual intervention. It is better suited for small, keyword-driven product list collection than continuous large-scale scraping.