News Crawler Engineering

Problem

News sources often mix list APIs, detail pages, anti-bot logic, CSR, and image loading. Treating a site as a one-off parser usually leads to missing original_source, storing column introductions as summary, loading all historical pages into memory, duplicate inserts, stopping pagination incorrectly when the API fails, and proxy blocks on overseas sites.

How It Works

This skill treats each site as one source in a long-running collection system:
- Recon first: confirm the real list API, pagination params, columnId/channel, and detail URL rule; verify title, timestamp, source, images, summary, and content.
- Choose mode: use API when it includes content; otherwise combine list API with detail pages, parse static HTML, or reverse signatures for strong anti-bot sites.
- Model first: keep lightweight fields in get_market_news_info and large fields such as content_html, images, and raw_data in get_market_news_content; use a business unique key rather than relying only on news_id.
- Extract content by priority: API blocks/paragraphs first, then scoped HTML selectors, then get_text(); separate paragraphs with \n\n and insert image placeholders at their positions.
- Incremental and backfill: derive thresholds from MAX(publish_time) per source and column, backfill 90 days without history, overlap 6 hours with history; dedupe by page, parse, upsert, and stop when a page is fully older than the threshold.
- Operational guardrails: write main and content rows in one transaction, persist failure status, alert on crash, zero success, and high failure rate, and bypass proxies explicitly for overseas endpoints.