AI Structured Data Analysis

Problem

Structured datasets can look ready for conclusions, but real CSV, Excel, Parquet, and JSON files often contain column shifts, encoding errors, missing values, skewed distributions, and small samples. Asking an agent to draw trends or run regressions directly can mistake silent parsing failures for business signals. The workflow needs to separate data validation, statistical computation, and interpretation so each step can be reviewed.

How It Works

The skill builds the process around pandas, scipy.stats, statsmodels, and numpy. It first inspects DataFrame shape, column dtypes, and head/tail samples. It then computes means, medians, standard deviations, skewness, kurtosis, and category counts. Trends are explored with rolling averages, seasonal decomposition, group-by aggregation, and pivot tables. Key claims rely on correlation matrices, t-tests, chi-square tests, and ANOVA, with p-values, confidence intervals, and effect sizes reported together. Outliers are checked with IQR and z-score, while missing values, skew, collinearity, and small samples have explicit handling.

Boundaries

It suits exploratory structured-data analysis and statistical reporting, not domain modeling by itself. If the target has more than 30% missing values, distributions are heavily skewed, predictors are highly collinear, or n < 30, the method should be downgraded. Mixed-type columns, non-stationary time series, and confounders should be flagged as limitations, avoiding the jump from statistical significance to business causality.