Scikit learn

scikit-learn Skill

General-purpose machine learning with scikit-learn. Covers unsupervised methods (clustering, GMM, PCA, t-SNE, UMAP, manifold learning, evaluation metrics), supervised methods (classification, prediction-focused regression via Ridge/Lasso/ensemble methods, model evaluation, cross-validation), and shared infrastructure (preprocessing, Pipeline construction, feature selection). Use when performing cluster analysis, dimension reduction, classification, prediction-focused regression, or model evaluation in Python. For econometric regression (OLS, FE, IV, DiD), see pyfixest and statsmodels skills instead.

Comprehensive skill for machine learning in Python with scikit-learn. Covers unsupervised methods (clustering, decomposition, manifold learning), supervised methods (classification, regression), and shared infrastructure (preprocessing, pipelines, evaluation). Use decision trees below to find the right guidance, then load detailed references.

What is scikit-learn?

scikit-learn is the standard general-purpose machine learning library for Python:

• Consistent API: Every estimator follows fit() / predict() / transform() — learn once, apply everywhere
• Unsupervised methods: Clustering (KMeans, DBSCAN, HDBSCAN, hierarchical), decomposition (PCA, NMF, SVD), mixture models (GMM), manifold learning (t-SNE)
• Supervised methods: Classification (logistic regression, random forest, gradient boosting, SVM) and prediction-focused regression (Ridge, Lasso, ensemble methods)
• Model evaluation: Cross-validation, grid search, metrics for both classification and clustering
• Pipelines: Chain preprocessing and models into reproducible, leak-free workflows

Version Notes

This skill targets scikit-learn 1.8.0. Notable changes in recent versions:

• HDBSCAN added as a first-class estimator (1.3+)
• set_output(transform="pandas") for DataFrame output from transformers (1.2+)
• HistGradientBoosting estimators are now stable (1.0+)
• n_init="auto" default for KMeans (1.4+) — uses 10 for init="random", 1 for init="k-means++"

How to Use This Skill

Reference File Structure

Each topic in ./references/ contains focused documentation:

File	Purpose	When to Read
quickstart.md	Import patterns, fit/predict/transform API, Pipeline, train_test_split	First use of scikit-learn
clustering.md	KMeans, AgglomerativeClustering, DBSCAN, HDBSCAN, SpectralClustering, OPTICS	Cluster analysis tasks
mixture-models.md	GaussianMixture, BayesianGaussianMixture, BIC/AIC model selection	Model-based clustering, soft assignments
decomposition.md	PCA, KernelPCA, TruncatedSVD, NMF, IncrementalPCA	Dimension reduction tasks
manifold.md	t-SNE, UMAP (umap-learn), Isomap, LLE, MDS, SpectralEmbedding	Visualizing high-dimensional data
evaluation-unsupervised.md	silhouette_score, Davies-Bouldin, Calinski-Harabasz, ARI, NMI, gap statistic	Validating cluster solutions
preprocessing.md	StandardScaler, encoders, ColumnTransformer, Pipeline construction	Preparing data for ML
classification.md	LogisticRegression, RandomForest, GradientBoosting, SVC, KNeighbors	Classification tasks
regression-ml.md	Ridge, Lasso, ElasticNet, tree/ensemble regressors, SVR	ML regression (prediction-focused)
evaluation-supervised.md	Accuracy, F1, ROC-AUC, confusion matrix, cross_val_score, GridSearchCV	Evaluating supervised models
feature-selection.md	SelectKBest, RFE, permutation_importance, VarianceThreshold	Selecting informative features
gotchas.md	Data leakage, scaling errors, t-SNE misinterpretation, class imbalance	Avoiding common mistakes
interpretation.md	SHAP values (TreeExplainer, KernelExplainer), permutation importance visualization, partial dependence plots, ICE plots	After training a model, when interpretation or explanation is needed
fairness.md	fairlearn MetricFrame, ThresholdOptimizer, ExponentiatedGradient, demographic parity, equalized odds	Assessing or mitigating fairness of supervised models

Reading Order

1. New to scikit-learn? Start with quickstart.md then the task-specific reference
2. Clustering task? Read clustering.md, then evaluation-unsupervised.md
3. Classification task? Read classification.md, then evaluation-supervised.md
4. Need preprocessing? Read preprocessing.md (covers Pipeline construction)
5. Having issues? Check gotchas.md first
6. Interpretation task? Read interpretation.md, then check supervised-ml.md in data-scientist skill for methodology
7. Fairness assessment? Read fairness.md, then check supervised-ml.md in data-scientist skill for conceptual framework

Related Skills

Skill	Relationship
data-scientist	Methodology guidance — load exploratory-unsupervised.md for "when and why" behind unsupervised methods
pyfixest	Econometric regression: OLS with fixed effects, IV, DiD, clustered SEs, hypothesis testing
statsmodels	Statistical modeling: OLS without FE, GLM, time series, diagnostic tests
polars	Data preparation before ML (convert to pandas/numpy before passing to scikit-learn)
geopandas	Spatial analysis — use geopandas for geographic data, not scikit-learn
plotnine	Custom visualization beyond scikit-learn's built-in plotting
data-scientist	Load supervised-ml.md for supervised ML methodology — the "when and why" behind prediction, interpretation, and fairness

Routing guidance:

• For econometric regression (hypothesis testing, standard errors, coefficient interpretation), use pyfixest or statsmodels — not scikit-learn
• For unsupervised methodology (when to cluster, how to validate, what to report), read exploratory-unsupervised.md in the data-scientist skill
• For spatial analysis, use geopandas
• For data manipulation, use polars

Quick Decision Trees

"I need to group observations" (Unsupervised)

What kind of data and clusters?
├─ Continuous data, roughly spherical clusters
│   ├─ Know k → KMeans (./references/clustering.md)
│   └─ Don't know k → try multiple k + silhouette/gap
│       (./references/clustering.md + ./references/evaluation-unsupervised.md)
├─ Continuous data, arbitrary shapes
│   ├─ Dense clusters, possible noise → DBSCAN or HDBSCAN (./references/clustering.md)
│   └─ Need soft assignments → GaussianMixture (./references/mixture-models.md)
├─ Need hierarchy / dendrogram → AgglomerativeClustering (./references/clustering.md)
├─ Mixed data types → Gower distance workaround (./references/gotchas.md)
└─ Need probabilistic model comparison → GaussianMixture with BIC
    (./references/mixture-models.md)

"I need to reduce dimensions" (Unsupervised)

What is the goal?
├─ Linear reduction for subsequent analysis → PCA (./references/decomposition.md)
├─ Large sparse data → TruncatedSVD (./references/decomposition.md)
├─ Non-negative components → NMF (./references/decomposition.md)
├─ Visualization of structure → t-SNE or UMAP (./references/manifold.md)
│   └─ CAUTION: visualization only, not for analysis
│       (see data-scientist exploratory-unsupervised.md for methodology)
├─ Nonlinear manifold learning → Isomap or LLE (./references/manifold.md)
└─ Correspondence analysis (CA, MCA) → use the prince library

"I need to predict a categorical outcome" (Supervised)

What constraints?
├─ Interpretable model needed → LogisticRegression or DecisionTreeClassifier
│   (./references/classification.md)
├─ Best predictive performance → GradientBoostingClassifier or RandomForestClassifier
│   (./references/classification.md)
├─ High-dimensional sparse data → LogisticRegression with penalty
│   (./references/classification.md)
├─ Small dataset, few features → KNeighborsClassifier or SVC
│   (./references/classification.md)
└─ Need probability estimates → any classifier with predict_proba()
    (./references/classification.md)

"I need to predict a continuous outcome" (Supervised)

What kind of regression?
├─ NOTE: For econometric regression (hypothesis testing, standard errors,
│   coefficient interpretation), use pyfixest or statsmodels instead
├─ Prediction-focused, nonlinear → GradientBoostingRegressor or RandomForestRegressor
│   (./references/regression-ml.md)
├─ High-dimensional with regularization → Lasso, Ridge, or ElasticNet
│   (./references/regression-ml.md)
├─ Nonlinear relationships → GradientBoostingRegressor or SVR
│   (./references/regression-ml.md)
└─ Simple baseline → Ridge (./references/regression-ml.md)

"I need to evaluate a model"

What kind of evaluation?
├─ Unsupervised (no ground truth)
│   ├─ Cluster quality → silhouette_score, Davies-Bouldin
│   │   (./references/evaluation-unsupervised.md)
│   ├─ Stability → Bootstrap + compare across resamples
│   │   (./references/evaluation-unsupervised.md)
│   └─ Against known labels → ARI, NMI
│       (./references/evaluation-unsupervised.md)
├─ Supervised classification
│   ├─ Balanced classes → accuracy + F1 (./references/evaluation-supervised.md)
│   ├─ Imbalanced classes → precision, recall, ROC-AUC
│   │   (./references/evaluation-supervised.md)
│   └─ Model selection → cross_val_score or GridSearchCV
│       (./references/evaluation-supervised.md)
└─ Supervised regression
    ├─ R-squared, RMSE, MAE (./references/evaluation-supervised.md)
    └─ Model selection → cross_val_score or GridSearchCV
        (./references/evaluation-supervised.md)

"I need to interpret or explain a model"

What kind of interpretation?
├─ Feature importance (global) → SHAP beeswarm/bar or permutation importance
│   (./references/interpretation.md)
├─ Single prediction explanation → SHAP waterfall or force plot
│   (./references/interpretation.md)
├─ Feature effect visualization → PDP or SHAP dependence plot
│   (./references/interpretation.md)
├─ Fairness across demographic groups → MetricFrame
│   (./references/fairness.md)
└─ CAUTION: feature importance ≠ causal importance
    (see data-scientist supervised-ml.md for methodology)

File-First Execution in Research Workflows

Important: In data research pipelines (see CLAUDE.md), scikit-learn analyses are executed through script files, not interactively. This ensures auditability and reproducibility.

The pattern:

1. Write ML code to scripts/stage8_analysis/{step}_{task-name}.py
2. Execute via Bash with automatic output capture wrapper script
3. Validation results get automatically embedded in scripts as comments
4. If failed, create versioned copy for fixes

Closely read agent_reference/SCRIPT_EXECUTION_REFERENCE.md for the mandatory file-first execution protocol covering complete code file writing, output capture, and file versioning rules. All ML scripts must follow the Inline Audit Trail (IAT) standard -- see agent_reference/INLINE_AUDIT_TRAIL.md. For ML code, document model selection rationale (why this algorithm, why these hyperparameters, what assumptions) with # INTENT:, # REASONING:, and # ASSUMES: comments.

See:

• agent_reference/WORKFLOW_PHASE4_ANALYSIS.md -- Stage 8 (Analysis & Visualization)
• agent_reference/INLINE_AUDIT_TRAIL.md -- IAT documentation standard

The examples below show scikit-learn syntax. In research workflows, wrap them in scripts following the file-first pattern.

---

Quick Reference

Essential Imports

import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

The fit/predict/transform Pattern

# Supervised: fit + predict
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Unsupervised: fit + transform (or fit_transform)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Clustering: fit + labels_
kmeans.fit(X)
labels = kmeans.labels_

Common Operations

Operation	Code
Train-test split	X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
Scale features	X_scaled = StandardScaler().fit_transform(X)
Build pipeline	pipe = make_pipeline(StandardScaler(), LogisticRegression())
Cross-validate	scores = cross_val_score(model, X, y, cv=5, scoring="accuracy")
Grid search	GridSearchCV(model, param_grid, cv=5, scoring="accuracy")
KMeans clustering	KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
PCA	PCA(n_components=5).fit_transform(X_scaled)
Logistic regression	LogisticRegression(max_iter=1000).fit(X_train, y_train)
Random forest	RandomForestClassifier(n_estimators=100, random_state=42).fit(X_train, y_train)
Gradient boosting	HistGradientBoostingClassifier().fit(X_train, y_train)
Classification report	print(classification_report(y_test, y_pred))
Confusion matrix	confusion_matrix(y_test, y_pred)
Silhouette score	silhouette_score(X, labels)
Feature importance	model.feature_importances_
Permutation importance	permutation_importance(model, X_test, y_test, random_state=42)
Set output format	model.set_output(transform="pandas")

Topic Index

Topic	Reference File
Installation and imports	./references/quickstart.md
fit/predict/transform API	./references/quickstart.md
Pipeline construction	./references/quickstart.md
Train-test split	./references/quickstart.md
Reproducibility (random_state)	./references/quickstart.md
KMeans, MiniBatchKMeans	./references/clustering.md
AgglomerativeClustering	./references/clustering.md
DBSCAN, HDBSCAN, OPTICS	./references/clustering.md
SpectralClustering	./references/clustering.md
GaussianMixture	./references/mixture-models.md
BayesianGaussianMixture	./references/mixture-models.md
BIC/AIC model selection	./references/mixture-models.md
Soft cluster assignments	./references/mixture-models.md
PCA, KernelPCA	./references/decomposition.md
TruncatedSVD (sparse data)	./references/decomposition.md
NMF	./references/decomposition.md
IncrementalPCA	./references/decomposition.md
t-SNE	./references/manifold.md
UMAP (umap-learn)	./references/manifold.md
Isomap, LLE, MDS	./references/manifold.md
silhouette_score	./references/evaluation-unsupervised.md
Davies-Bouldin, Calinski-Harabasz	./references/evaluation-unsupervised.md
Adjusted Rand Index, NMI	./references/evaluation-unsupervised.md
Gap statistic	./references/evaluation-unsupervised.md
StandardScaler, MinMaxScaler	./references/preprocessing.md
OneHotEncoder, OrdinalEncoder	./references/preprocessing.md
ColumnTransformer	./references/preprocessing.md
Pipeline, make_pipeline	./references/preprocessing.md
LogisticRegression	./references/classification.md
DecisionTreeClassifier	./references/classification.md
RandomForestClassifier	./references/classification.md
GradientBoostingClassifier	./references/classification.md
SVC, KNeighborsClassifier	./references/classification.md
Ridge, Lasso, ElasticNet	./references/regression-ml.md
RandomForestRegressor	./references/regression-ml.md
GradientBoostingRegressor	./references/regression-ml.md
SVR, KNeighborsRegressor	./references/regression-ml.md
accuracy, precision, recall, F1	./references/evaluation-supervised.md
ROC-AUC, confusion matrix	./references/evaluation-supervised.md
cross_val_score, GridSearchCV	./references/evaluation-supervised.md
learning_curve	./references/evaluation-supervised.md
SelectKBest, RFE	./references/feature-selection.md
feature_importances_	./references/feature-selection.md
permutation_importance	./references/feature-selection.md
Data leakage	./references/gotchas.md
Scaling for distance-based methods	./references/gotchas.md
t-SNE/UMAP distance interpretation	./references/gotchas.md
Class imbalance	./references/gotchas.md
random_state reproducibility	./references/gotchas.md
SHAP values (TreeExplainer, KernelExplainer)	./references/interpretation.md
Permutation importance visualization	./references/interpretation.md
Partial dependence plots (PDP)	./references/interpretation.md
ICE plots	./references/interpretation.md
Model interpretation caveats	./references/interpretation.md
fairlearn MetricFrame	./references/fairness.md
ThresholdOptimizer	./references/fairness.md
ExponentiatedGradient	./references/fairness.md
Demographic parity	./references/fairness.md
Equalized odds	./references/fairness.md
LightGBM (LGBMClassifier, LGBMRegressor)	./references/classification.md, ./references/regression-ml.md
XGBoost (XGBClassifier, XGBRegressor)	./references/classification.md, ./references/regression-ml.md

Citation

When this library is used as a primary analytical tool, include in the report's Software & Tools references:

Pedregosa, F. et al. (2011). "Scikit-learn: Machine Learning in Python." Journal of Machine Learning Research, 12, 2825-2830.

Cite when: scikit-learn is used for machine learning models, clustering, dimensionality reduction, or cross-validation central to the analysis. Do not cite when: Only used for a single preprocessing step (e.g., StandardScaler in a pipeline where the primary model is from another library).

For method-specific citations (e.g., individual algorithms or techniques), consult the reference files in this skill and agent_reference/CITATION_REFERENCE.md.