Exploratory data analysis
Skill Amey-Thakur/AI-SKILLS/skills/data-science/exploratory-data-analysis
Plug-and-play skills and prompts for every AI coding agent
npx -y skills add Amey-Thakur/AI-SKILLS --skill exploratory-data-analysisAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- 19 days oldThe repository was created 19 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
- 4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Explore a new dataset systematically to understand its shape, quality, and signal before modeling. Use when you first get a dataset and need to know what is in it, what is wrong with it, and what is worth pursuing.
SKILL.md
3.0 KB, as published. Nobody here has run it
Exploratory data analysis
EDA is the conversation you have with a dataset before trusting it. Skip it and you model on garbage, miss the leak, or chase the wrong signal. The goal is understanding: what is here, what is broken, and where the signal lives, grounded in looking rather than assuming.
Method
- Establish the shape first. Rows, columns, types, and what one row represents (the grain). Confirm the grain matches your belief; a dataset you think is one-row-per-user but is one-row-per-event breaks every later aggregation (see warehouse-modeling's grain rule).
- Profile every column. For numerics: distribution, range, mean/median, outliers. For categoricals: cardinality and value counts. For all: missingness rate and pattern. Summary statistics and a histogram per column surface the surprises (a "price" column with negatives, an "age" of 999, a category with 40% blanks) that would corrupt a model silently.
- Investigate missingness and outliers as signal. Is data missing at random, or does missing mean something (no purchase = null revenue)? Are outliers errors (a typo'd 10000 for 100) or real extremes? The answer changes how you handle them (see data-cleaning); do not blindly drop or impute before understanding why.
- Look at relationships, especially with the target. Correlations and cross-tabs between features, and between each feature and the target: what actually moves the outcome, what is redundant, what is surprising. Plot the strong relationships (see data-visualization); a scatter reveals what a correlation number hides (nonlinearity, clusters).
- Hunt for leakage and too-good signal. A feature that predicts the target almost perfectly is usually leakage (it encodes the answer, or comes from after the prediction moment), not a great feature. Suspiciously strong signal gets audited, not celebrated (see train-test-discipline, leaderboard-strategy).
- Write down what you found. The grain, the quirks, the columns to drop or fix, the hypotheses to test, the leakage risks. EDA that lives only in your head does not inform the team or your future self (see notebook discipline in data-storytelling).
Boundaries
- EDA describes what is in the data; it does not tell you the data is correct, representative, or collected without bias (see sampling and the correlation-vs-causation trap in correlation-causation).
- Looking at the target relationships on the full dataset risks biasing your choices; keep a held-out set untouched, and be aware that decisions made from EDA are a mild form of peeking (see train-test-discipline).
- EDA is iterative and open-ended; time-box it to the decision it serves, or it expands indefinitely (see research-planning's depth budget).