Reference Harvester: Auto PDF & Citation Finder

Problem

When managing papers or references, PDF reference sections often appear as numbered lists, author-first entries, or URL-only citations. Text extraction can also split https://, DOIs, and paths with spaces, producing unusable values like doi.org/ 10.1016/.... A simple regex pipeline may then yield truncated titles, missing authors, invalid BibTeX, and HTML error pages saved as PDFs.

How It Works

Reference Harvester turns the process into scripted steps: it extracts full text with pdfplumber, then parses References, 参考文献, or Bibliography sections using numbered and author-first reference patterns. It repairs broken URLs, removes page markers, headers, and footnote artifacts, and emits JSON with index, title, url, doi, arxiv_id, and raw. It then enriches each item via DOI or URL lookup to recover authors, journal, volume, and pages, and generates BibTeX entries as article for journal papers or unpublished for preprints. For downloads, it prioritizes arXiv, then attempts direct PDFs, PMC/EuropePMC, MDPI, Nature/Springer OA, PLoS, RSC, Wiley, and Elsevier endpoints, validating files by checking the %PDF- magic bytes.

Limitations

The skill is best for extracting references from a single paper PDF, not for resolving inline footnotes or paywalled items. Some publishers return 403 even for open-access papers, so manual download may still be needed. For nonstandard styles such as Vancouver or inline citations, the extracted text may need inspection and heuristic adjustments.