Douyin Hot Video Data Scraper

What Problem It Solves

Many analytics and content-ops workflows need fast access to Douyin hot videos, e-commerce promotions, and real-time Baidu trends, but these pages are often SPAs with dynamic rendering and messy author fields. Direct HTTP requests to Chanmama ranking pages can return only an empty shell. Manual copying or ad hoc scripts often miss details such as iesdouyin link conversion, follower-count cleanup, and category tagging. This skill turns scraping, cleaning, and reporting into a reusable pipeline for TOP 10 data.

How It Works

• Data capture: Uses Selenium to render Chanmama’s hot video and e-commerce promotion rankings, extracting title, creator, likes, comments, and plays from DOM tables. For Baidu real-time trends, it parses embedded JSON from server-rendered HTML with regex for word and hotScore.
• Data cleaning: Converts hidden share links into canonical douyin.com/video/{ID} URLs and removes noisy follower counts and timestamps from creator names using regex cleanup.
• Classification and output: Labels videos into 18 content categories or e-commerce products into 7 product categories, then generates a four-section text report and timestamped CSV files to reduce file locking conflicts.

Boundaries and Caveats

It is better suited for lightweight trend snapshots than large-scale collection. Chanmama is an SPA, so DOM or URL changes can break selectors; Baidu’s embedded JSON structure may also change. Frequent access can trigger anti-bot challenges, so request frequency should be controlled. The runtime depends on Python, Chrome, and a matching ChromeDriver, and headless mode may render blank on some systems, where headed debugging can help.