Article Fetching Crawler

Problem

Chinese policy and news pages are scattered across multiple government portals and news sites. Articles about AI, digitalization, and informatization are often mixed with procurement notices, public announcements, and administrative documents. Manually finding relevant pieces requires repeated site visits, keyword searches, noise filtering, link collection, and summary writing. This skill turns that workflow into a repeatable crawl and filtering process.

How It Works

• Source coverage: supports the National Energy Administration, Ministry of Industry and Information Technology, Cyberspace Administration of China, National Development and Reform Commission, Sichuan Provincial Department of Economy and Information Technology, State Informatization Expert Advisory Committee, Xinhuanet, People's Daily Online, CCTV News, Guancha, and The Paper.
• Keyword filtering: prioritizes topics such as artificial intelligence, digitalization, informatization, AI, algorithm, big data, and cloud computing, while excluding noise terms such as procurement, announcement, tender, bid winning, and contract.
• Output structure: writes original_articles/ for article text, article_links/ for links and metadata, ai_summaries/ for summaries, and crawled_links.json for deduplication tracking.
• Runtime behavior: includes automatic deduplication, publication-date sorting, incremental file indexing, AI summarization, error handling, and retry mechanisms.

Boundaries

It is better suited for collecting Chinese government and news articles than for general-purpose full-web crawling. Adding new sources requires extending crawler classes based on BaseCrawler and maintaining site-specific structure. Keyword filtering is heuristic, so edge cases may still be missed or incorrectly included.