Data Cleaning & Analysis

What Problem It Addresses

When tabular data arrives, the risk is rarely one isolated cleaning step. It is order: deleting missing values too early can remove meaningful business signals, ignoring encoding can misread CSV files, and skipping distribution checks can lead to wrong imputation. This skill makes the diagnostic sequence explicit: load, inspect, clean, explore, analyze, then report, so analysis does not start on unexamined data.

How the Pipeline Works

It runs LOAD → CLEAN → EXPLORE → ANALYZE → REPORT, where each stage produces input for the next. Cleaning handles missing values, duplicates, IQR-based outliers, and type conversions. Missing-rate rules separate >50%, 5%-50%, and <5%; skewed numeric columns favor median, while highly correlated pairs receive deletion suggestions. The final summary includes a 0-100 quality score, operations performed, skew/correlation/cardinality findings, and feature suggestions such as log/sqrt, encoding, or interactions.

Boundaries and Caveats

It is suited to structured tabular preprocessing where a repeatable audit trail matters, but it does not replace domain judgment. Extreme values may be valuable business signals, so IQR should be treated as a prompt. Training and test handling should follow project policy, and encoding, units, and primary-key semantics still need manual confirmation.