Web Novel Scraper and Reading Manager

Problem

Many web-novel helpers stop at book discovery or bookmarking, while the harder part remains: fetching readable chapters, tracking updates, and exporting them into offline formats. webnovelscraper treats this as a local data pipeline: find a book, register it, crawl chapter text, then export EPUB/TXT or generate MP3 audio, with state kept in SQLite and chapter files on disk.

How It Works

The main workflow uses add-novel, crawl, download, and tts. Adding a book stores metadata such as title, author, platform, and catalog URL without downloading content. Crawling fetches the catalog, lists chapters, downloads selected chapter text, removes watermark and navigation noise, and saves files under ~/Novels/<title>/chapters/. Reading, export, and TTS consume those saved files, so interrupted runs can resume and missing chapters can be backfilled.

• Progress handling is safer: incremental updates only advance to the largest contiguous completed chapter, so failed chapters are retried instead of skipped.
• Performance is practical: v2.1.0 uses a thread pool with default concurrency of 3, while keeping per-chapter delays.
• Export is cleaner: EPUB output escapes HTML in titles and paragraphs, reducing corruption from special characters.
• TTS is more stable: long chapters are split at sentence boundaries, retries use exponential backoff, and incomplete audio is removed.

Boundaries

It works best with free chapters from aggregator-style sites such as Biquge clones, especially for personal offline reading, library organization, and audio listening. Heavily JS-rendered or paywalled platforms like Qidian and Fanqie are less direct and may require Playwright or CDP browsers, disabled by default. TTS depends on edge-tts and may need a proxy in mainland China. Use it only for personal reading and avoid aggressive scraping or redistribution.