Ml data leakage guard

Skill toilact/godchi/skills/ml-data-leakage-guard

Use when writing or editing feature engineering, dataset building, label/target creation, scalers/encoders, or any train/serve pipeline — especially time-series or trading features. Catches lookahead/target leakage (e.g. close.shift(-1)), future joins, and fit-on-full-data before they poison a model.From its SKILL.md

Install

npx -y skills add toilact/godchi --skill ml-data-leakage-guard

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

3.2 KB, as published. Nobody here has run it

ML Data-Leakage Guard

Overview

A feature for time T may use ONLY information available at decision time T. The label is the only thing allowed to look into the future. Leakage makes offline metrics look great and production fail — one of the most damaging silent ML bugs (especially in time-series / trading).

When to Use

Trigger on any of: building/editing a feature builder, creating labels/targets, fitting a scaler/encoder/imputer, joining external data (news, prices, sentiment), resampling, rolling windows, shift, rolling, groupby().transform, train/val/test prep.

Core rule (before → after)

# ❌ LEAK: feature uses tomorrow's price as today's input
features["close_next"] = df["close"].shift(-1)        # negative shift = future
features["ret_fwd"]    = df["close"].pct_change().shift(-1)

# ✅ OK: feature uses only data known at decision time T; label is the future target
features["close_t"]    = df["close"]                  # T data
features["ret_5d"]     = df["close"].pct_change(5)    # past returns (>=0 shift)
label                  = df["close"].shift(-1)        # the TARGET only, never a feature

Checklist (run mentally on every feature)

Negative shift / forward window? Any shift(-n), .shift(-1), rolling(...).mean().shift(-1), [t+1] indexing in a feature → leak. Allowed only when assigning the label.
Timestamp boundary. Does each feature only use rows with timestamp ≤ the decision cutoff (for a daily model, e.g. ≤ market close of day T)? External data (news, sentiment, late-arriving records) dated after the cutoff must not enter feature(T).
Fit scope. scaler.fit / encoder.fit / imputation stats / target-encoding computed on the train fold only, then .transform on val/test — never fit on the full dataset.
Aggregates across the whole series (global mean, min-max, target mean per category) computed without a time boundary → leak.
Duplicate/identity columns that are a transform of the label (label = round(feature)).

Common mistakes

Mistake	Why it leaks	Fix
`shift(-1)` in features	uses T+1 data	shift ≥ 0 for features; negative only for label
`StandardScaler().fit(X)` then split	val/test stats bleed into train	fit on train fold only
join news on calendar date	news published after cutoff counts as "today"	filter to publish_ts ≤ cutoff
`df.fillna(df.mean())` before split	mean uses future rows	impute inside the fold

Red flags — STOP

"It's just a quick feature, I'll check leakage later"
"The backtest accuracy jumped to 90%" (almost always leakage)
Fitting any transform before the time-based split
Any -1/forward reference in a column that feeds the model

When in doubt: state the decision cutoff explicitly and prove every feature respects it. Pairs with walk-forward-guard. Reply in Vietnamese.