Document Duplicate Cleaner

Problem

Knowledge directories often accumulate copies of the same PDF, Word, Excel, or Markdown document after downloads, syncs, backups, or renames. File names are unreliable: Contract_v2, Contract_copy, and contract_final may contain identical bytes, while files with the same name may differ. Manually opening each document to compare is slow and increases the risk of deleting the wrong version.

How It Works

Document Duplicate Cleaner compares document content instead of relying on file names. The main workflow is:

• Directory scan: Walks the target directory recursively, skips common directories such as .git, node_modules, and __pycache__, and supports regex-based --exclude patterns for files or directories.
• Hashing: Reads .pdf, .docx, .xlsx, .pptx, .txt, .md, .csv, .json, and .xml files in chunks and computes their SHA-256 digest, avoiding loading large documents fully into memory.
• Grouped report: Files with the same hash are grouped together. The report lists group size, file size, modification time, and relative path. For text-based files such as .txt, .md, .csv, .json, .xml, .html, and .eml, it can also show a short preview to help confirm the content.
• Safe cleanup: The default dry-run mode only lists files that would be removed and the estimated space savings. Deletion happens only after --delete, with interactive confirmation and options to skip a group, delete all duplicates, or select specific files by number. The earliest modified file is treated as the original copy to keep.

Caveats

This tool is best suited for local directory cleanup of identical document copies, not for deduplicating across databases, cloud storage servers, or email systems. Binary formats such as .pdf and .docx can only be matched by content hash, so version differences are not explained. Text previews are only a convenience check. If the directory contains symlinks, restricted permissions, or unusual encodings, review the report first and confirm that the file you keep is the correct version.