Data science

Defaults

• Python preferred (R only if explicitly requested)
• Libraries: Polars (preferred; fast, typed, pandas-compatible API), NumPy, SciPy, Scikit-Learn, XGBoost, Seaborn, Matplotlib
• Environment: uv for deps, ruff/pyright/pytest

Analytical code style

Taste rules for data-shaped code (pipelines, caches, scoring paths) — apply when writing AND when reviewing:

• DataFrames over parallel collections. A dict of arrays, a list of dicts, or an out.append(...) loop is a latent DataFrame. Use Polars: same-length and per-column dtype guarantees for free, inspectable (sort/filter without loops), and well-typed for checkers.
• One dataframe library per codebase. Polars preferred over pyarrow over pandas; never mix them. Polars covers parquet IO including file metadata (write_parquet(metadata=...), read_parquet_metadata).
• Named algorithms over hand-rolled math. If sklearn/scipy ships the operation, call it — NearestNeighbors(...).kneighbors returns sorted distances + indices and replaces matmul + argpartition cleverness a reviewer must re-derive; same for haversine_distances over inlined trig. (At high dimension use algorithm="brute"; tree indexes die past ~20-D.)
• Store data in the shape its consumers read. A loader that reshapes on every read means the storage format is wrong; adapter functions that exist only to convert shapes are a smell.
• One machine-readable output. The data artifact (parquet) is the machine interface; CLI/stdout is for humans. No parallel JSON paths through the same code.
• Fewest artifacts. One file holding everything beats synchronized caches; only cache what is expensive to recompute (embeddings, not metadata).

Modeling

• Default model: XGBoost with tree_method="hist", low learning rate, early stopping
• Validation: Nested — 20% holdout + 10-fold CV on training set
• Feature importance: Use pred_contribs=True in XGBoost (NOT shap package)

Prohibitions

• No shap package — use XGBoost's built-in pred_contribs=True
• No SVMs (don't scale)
• No neural nets unless explicitly requested
• No TensorFlow (use PyTorch if needed)
• No Plotly (use Seaborn/Matplotlib)
• No SMOTE or upsampling (use scale_pos_weight or sample_weight)

EDA Checklist

Numeric columns: count, missing %, min/max, mean/median, quantiles (5/25/50/75/95), std Categorical columns: cardinality, top 5/bottom 5 values, missing % Text columns: cardinality, length distribution (min/median/max chars) Target: distribution, class balance (if classification)

Feature Engineering

• Categoricals: OrdinalEncoder (not one-hot for tree models)
• Rates (denominator varies, near-zero to huge): empirical-Bayes shrink toward the global mean — (k + α·μ)/(n + α) — never hard-cutoff small-denominator groups. CI lower bounds (Poisson/Wilson) are a valid fallback, but prefer EB shrinkage first: simpler, and it forgives small samples without dropping rows.
• High-cardinality target encoding (modeling): smooth toward the global mean (empirical Bayes) and/or leave-one-out encode and/or add train-only noise — Owen Zhang's classic XGBoost categorical recipe. LOO + noise guard against target leakage/overfit when fitting a model; for description or ranking, EB shrinkage alone suffices.
• Text: sentence-transformers embeddings preferred, TF-IDF as fallback
• Dates: extract year, month, day_of_week, is_weekend, days_since_reference
• Missing: let XGBoost handle natively (don't impute for tree models)

Metrics

Regression: RMSE + R-squared + FVE (fraction of variance explained) Classification: log loss + confusion matrix + FVE-binomial Unsupervised: Calinski-Harabasz or Davies-Bouldin for cluster quality

Unsupervised

• PCA / TruncatedSVD for dimensionality reduction
• K-Means++ for clustering
• Always report explained variance ratio for PCA

Artifacts

Save all outputs to report_artifacts/:

• Model files, feature importance plots, validation curves, confusion matrices
• Produce Quarto or Pweave reports when applicable