Real-time China Trending Data Collector

The Problem to Solve

When building data analysis applications, sentiment monitoring systems, or content recommendation engines, developers often face a core challenge: how to efficiently and stably acquire real-time trending data from various Chinese internet platforms. Manually visiting platforms like Weibo, Douyin, or Bilibili and parsing their pages is not only time-consuming but also prone to failure due to frequent page structure changes or anti-scraping measures. You need a structured data source to directly integrate these trending metrics (like keywords and hot values) into downstream analysis pipelines.

How the Skill Works

This skill provides a set of Node.js scripts for real-time data scraping from specified platforms. Its core capabilities include:

• Multi-faceted Data Collection: Covers four major categories of data sources.
• crawl-hot.js: Scrapes real-time hot search rankings from platforms such as Douyin, Weibo, Baidu, Bilibili, and Kuaishou.
• crawl-music.js: Fetches hot and rising song charts from QQ Music, NetEase Cloud Music, etc.
• crawl-entertainment.js: Collects Maoyan movie box office, TV show ratings, and App Store app/game rankings.
• crawl-paper.js: Retrieves PDF download links and high-resolution images for People's Daily e-paper editions.
• Unified Output Format: All scripts output structured JSON with clearly defined fields, such as word (keyword) and hot_value (hot value) for hot searches, or name (song title) and geshou (singer) for music. This eliminates the need for consumers to parse messy HTML.
• Flexible Parameterized Invocation: Precisely control scraped content via command-line parameters. For example:
• Specify platform: node scripts/crawl-hot.js --platform=weibo
• Specify type: node scripts/crawl-entertainment.js --type=movie
• Specify date: node scripts/crawl-paper.js --date=2025-03-20

The key workflow involves choosing the corresponding script and parameter combination based on user intent (e.g., “show me today's Weibo hot searches”) within the skill's installation directory, executing the node command, and parsing the returned JSON object to obtain the desired data.

Applicable Boundaries & Considerations

Please note the following technical details before use:

• Runtime Environment: Node.js 18+ must be installed, as the scripts rely on the built-in fetch API.
• Data Real-time & Stability: Data is scraped directly from upstream platforms, with a typical response time of 1-3 seconds. The Kuaishou platform's hot search API may be unstable at times, occasionally returning empty data.
• Working Directory: Scripts should be executed with the current working directory set to the skill's root directory to ensure relative path references within scripts resolve correctly.
• No API Keys Required: All scraping functionalities work out-of-the-box without any API Key or secret configuration.
• Output Handling: The returned JSON contains fields ready for immediate use (e.g., image_url for newspaper pages), which can be directly used for Markdown image rendering or further data analysis.