Douyin Knowledge Collector

Problem

Collecting industry knowledge from Douyin is often fragmented: videos contain practical opinions, but content is hard to search, transcribe, and organize. Manually downloading, converting audio, and cleaning Whisper output creates repetitive work, especially when Chinese transcription lacks punctuation, mixes traditional characters, or contains homophone errors. This skill provides a repeatable pipeline for engineers and analysts: input keywords, output transcripts and structured source material for a knowledge base.

How it works

The core flow is: Chrome CDP search → video URL extraction → audio or video download → Whisper transcription → quality reports and knowledge-base drafting. Key steps:

• Search and collection: connects to a logged-in Chrome session via Chrome DevTools Protocol, takes first-page results, uses random pauses, video intervals, and CDP retry behavior to reduce instability.
• Audio extraction: prefers DASH audio streams, falls back to full video when needed, and supports resumable downloads with retries for network interruptions.
• Transcription and cleanup: uses openai-whisper for Chinese transcription, guides punctuation output with initial_prompt, and applies a built-in traditional-to-simplified mapping.
• Quality grading: generates quality_report.json, classifying transcripts as high, medium, or low so longer and more usable outputs can be prioritized.
• Knowledge drafting: merges texts under transcripts/, removes duplicates, groups content by topic, and produces raw and refined knowledge-base files.

Boundaries

The skill is designed for Douyin only and does not support Bilibili, Xiaohongshu, or YouTube. It requires a logged-in Chrome session, Python, ffmpeg, and whisper. Whisper Chinese accuracy is about 85%, so final knowledge-base content still needs human review for homophone errors, domain terms, numbers, and uncertain items.