Chinese News Digest

Problem

Chinese news feeds are fragmented, and individual articles often mix clickbait, tutorials, promotional copy, and local social news. If you track policy, industry, or technology topics daily, manual reading is expensive because it also requires handling duplicate reporting, expired dates, and incomplete sentences.

How It Works

This skill treats the workflow as a practical pipeline:
- It uses SQLite to maintain tables such as monitor_websites, system_keywords, and articles, scraping pages with CSS selectors and storing normalized records.
- It removes near-duplicates with Jaccard 2-gram similarity, and cross_day_dedup.py compares recent digests to reduce cross-day repeats.
- A rule engine filters tutorials, marketing copy, entertainment, violence, and education content, then scores paragraphs by position, numeric density, signal words, and entity density to preserve complete sentences.
- If NEWS_DIGEST_LLM_API_KEY is configured, it can call an LLM for a second pass; otherwise that stage is skipped and rule-based summaries are used. The output is a readable digest, not a simple full-text dump.

Limits

It is best for topic-driven Chinese news digests, not as a general search engine. Anti-scraping, SSL issues, or layout changes can cause some sources to fail, and the database grows over time, so old rows need periodic cleanup. Scrape results depend on page structure, so keep failure logs.