AI News Known: Intelligent News Collection and Deduplication

Problem

Daily AI news is scattered across hot lists, newsletters, open-source releases, and papers. Manual filtering creates duplicate pushes and noise, making it hard to keep a readable digest. ai-news-known targets recurring AI news workflows that need a concise, archive-friendly briefing.

How It Works

• Multi-source collection: a script aggregates Chinese hot lists from Weibo, Zhihu, Baidu, Toutiao, Wall Street CN, The Paper, and optional platform APIs.
• AI filtering: high-confidence terms such as LLM, Agent, RAG, and 大模型 are prioritized; medium-confidence terms like chips, compute, and fine-tuning are evaluated with context.
• Deduplication memory: it reads data/history.json, compares title hashes and similarity scores, and filters repeated items or reworded versions of the same event.
• Search supplementation: an agent can use web_search to add AI-specific news missed by hot lists, then apply the same filtering and deduplication.
• Output: the default is a dark-theme HTML page; Markdown, compact, and summary formats are also available.

Boundaries

It is suitable for a Chinese hot-list plus search-based AI digest, not deep research. If all sources fail, it falls back to web_search; screenshots and Xiaohongshu copy are only generated on explicit request. History records are cleaned automatically.