Multi-Source Data Cleaner Pro

Problem it solves

When teams deliver CSV, Excel, JSON, Parquet, or SQL dumps that must be combined into one analytical table, the bottleneck is often not reading files but reconciling inconsistent column names, mixed encodings, varied date and boolean formats, different missing-value semantics, ambiguous duplicates, and inconsistent PII handling. multi-source-data-cleaner-pro targets this batch preprocessing task and turns ad hoc cleaning rules into an auditable pipeline.

How it works

• Source profiling: detects format, encoding, delimiter, header row, and infers column types, null rates, and cardinality.
• Type normalization: converts thousands separators, scientific notation, and currency symbols to numeric values; normalizes many date formats to ISO 8601; standardizes booleans, phone numbers to E.164, Chinese full/half-width names, and ID prefixes.
• Missing-value handling: applies column-level drop_row, mean/median/mode imputation, constant fills, forward_fill, or interpolation, and writes an _imputed flag.
• Schema reconciliation: aligns columns by exact match, fuzzy match, and type compatibility, producing crosswalk.json; users can override with mapping.yaml.
• Fuzzy deduplication: uses blocking and record linkage for names, addresses, and IDs, with lower-confidence merges routed for human review.
• Quality scoring: rates completeness, accuracy, consistency, timeliness, uniqueness, and validity from 0 to 100 with drill-down details.

Boundaries

It is intended for offline batch work on structured or semi-structured tabular data, not free-text extraction, proprietary binary formats without parsers, or real-time stream cleaning. PII is masked by default, original files are left untouched, and processing stays local.