Data Cleaning & Format Conversion Tool

Problem It Solves

Data often becomes difficult to use before analysis or modeling begins: CSV, JSON, XML, YAML, Excel, and TSV files are mixed together, headers are inconsistent, and dirty rows contain nulls, duplicates, malformed values, encoding issues, and inconsistent formats. Manual cleanup is easy to repeat wrong, while ad hoc scripts usually fit only one file shape. This skill turns common messy-data problems into a reusable set of cleaning and conversion operations.

How It Works

It is organized around a file-in, rule-processing, output workflow. Key capabilities include:

• Format conversion: move structured data between CSV, JSON, XML, YAML, Excel, and TSV.
• Data cleaning: handle duplicates, null values, abnormal-value repair, and format normalization.
• Encoding repair: detect garbled text and convert between encodings such as UTF-8, GBK, GB2312, and Latin1.
• Regex processing: extract, replace, or split column values using regular expressions.
• Column operations: rename, map, split, or merge columns and apply type conversions.
• Data validation: check email addresses, phone numbers, ID numbers, addresses, and other field formats.
• Batch processing: convert and clean files at the directory level, useful for many files with similar structures.

Boundaries and Notes

This skill fits local file or directory data cleanup, format unification, and field standardization, especially when messy files need to become usable tabular data before analysis. It is not a real-time pipeline, a full ETL orchestrator, or a database governance system. If the task includes streaming data, transactional database operations, cross-system synchronization, or access control, use this as one processing step in a larger pipeline. For sensitive fields, define masking and retention rules before running batch cleanup.