Local Multi-Document Similarity Analyzer

Problem

In bidding, proposal review, and document compliance workflows, the hard part is usually not “whether text is duplicated,” but where it is duplicated, which file it resembles, and how severe the overlap is. Whole-document similarity scores can be misleading because headings, numbering, and short boilerplate phrases may look similar even when the main body is original. Manual paragraph-by-paragraph comparison becomes difficult once the corpus includes multiple .docx, .pdf, .txt, or .md files. Local Multi-Document Similarity Analyzer is built for this cross-document comparison problem: it aims to produce paragraph-level, reviewable results instead of a single opaque score.

How It Works

The skill focuses on body-text detection, paragraph alignment, similarity judgment, and report generation:

• Inputs: it accepts at least 2 files and can process Word, PDF, plain text, and Markdown documents.
• Body filtering: chapter numbers, short headings, and template-like fragments are identified and filtered to reduce noise from non-content text.
• Paragraph normalization: overly short fragments may be merged with surrounding context to avoid false positives from isolated phrases.
• Similarity scoring: paragraph similarity is estimated using TF-IDF vectorization and related heuristics, with dynamic thresholds such as a higher threshold for short text and a lower threshold for longer text.
• Report output: it generates an HTML report with overall statistics, duplication progress indicators, paragraph-pair distributions, status judgments, and side-by-side comparison of duplicated paragraphs.
• Revision suggestions: optional rewrite guidance can be generated, but it is advisory only and is not written back into the source files.

The report classifies results by body-text duplication rate: below 10% is treated as passing, 10%~30% as warning, and 30% or above as failing. These thresholds are best used as a triage signal rather than a final compliance decision.

Boundaries and Caveats

This skill is suited to local, offline analysis of bidding documents, technical proposals, specification files, or compliance materials. Files are read locally by default and are not uploaded to external services, and the source files are not modified. First use may require installing Python dependencies such as python-docx, pdfplumber, scikit-learn, and Jinja2.

Results depend on text extraction quality, especially for scanned PDFs, complex tables, image-based text, or heavily customized layouts. The duplication thresholds are engineering heuristics; formal tender decisions should still be reviewed by the responsible business owner. Rewrite suggestions are editorial aids, not legal or compliance opinions.