agentsclimarketplace

Walkforward audit

Skill mospira/walkforward-audit

Audits time-based machine learning backtests, walk-forward validation, rolling retraining, forecasting evaluations, and temporal train/test pipelines for data leakage, faulty split logic, invalid feature timing, target leakage, calibration/tuning leakage, and misleading experiment comparisons.

Install
npx -y skills add mospira/walkforward-audit

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 9 stars9 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Audits time-based machine learning backtests, walk-forward validation, rolling retraining, forecasting evaluations, and temporal train/test pipelines for data leakage, faulty split logic, invalid feature timing, target leakage, calibration/tuning leakage, and misleading experiment comparisons. Use when agent needs to review or change walk-forward code, time-series ML experiments, trading backtests, forecasting notebooks, rolling CV logic, or any model evaluation where past-to-future chronology matters.

SKILL.md

6.8 KB, as published. Nobody here has run it

Walk-Forward Leakage Auditor

Core Principle

Before trusting reported metric, state the temporal contract:

For a prediction made at <decision_time>, allowed inputs are <data sources> as known no later than <cutoff_time>. The target covers <future horizon>. Training may use rows ending at <train_end>, validation/calibration may use rows <validation_window>, and final reported performance comes only from <test_window>.

If the project cannot state this contract, the evaluation is not yet auditable.

Audit Workflow

  1. Map the evaluation timeline.

    • Identify the prediction timestamp, label horizon, feature cutoff, train window, validation/calibration window, test/scoring window, embargo/gap, and retrain cadence.
    • Verify that every reported metric is computed only after the model, thresholds, transforms, and selection rules are frozen for that test window.

  2. Trace data through the pipeline.

    • Follow raw data into features, labels, datasets, splits, fitted transforms, models, predictions, decisions, and metrics.
    • Verify that every fitted object is trained inside the fold/window where it is used.
    • Distinguish label construction from model features: label code may look forward, but label columns must be excluded from model inputs.

  3. Inspect code and artifacts against the checklist below.

    • For each suspicious pattern, decide: leak, harmless by construction, or needs a test/guard.
    • Prefer concrete file/line evidence over broad claims.

  4. Require a failable guard before trusting the result.

    • Add assertions for split ordering, no overlap, embargo, feature availability cutoff, fitted-transform scope, and comparable run controls.
    • Prefer a tiny synthetic fixture that fails if a future value leaks into a past prediction.
    • Rerun affected results after fixing leakage or faulty logic.

Split Logic Checklist

Check:

  • Train rows end before validation/test rows begin.
  • Embargo/gap is at least as long as the label horizon or any overlapping feature lookback that can leak target information.
  • Fold boundaries are based on event/prediction time, not row order after arbitrary filtering.
  • Same entity/event cannot appear in both train and test when that creates leakage.
  • Hyperparameter, threshold, feature-selection, and architecture choices do not use final test metrics.

Red flags:

  • train_test_split, KFold, StratifiedKFold, .sample(frac=...), or shuffle=True.
  • Selecting "best" model/threshold by test Sharpe, test AUC, test return, or final leaderboard score.
  • Reusing the same validation slice for early stopping, calibration, feature selection, and threshold selection without acknowledging optimism.

Feature Timing Checklist

Check:

  • Rolling, expanding, rank, z-score, and aggregation features are lagged to the prediction cutoff.
  • Cross-sectional features only use constituents and data known at the prediction time.
  • Joins to fundamentals, events, news, status, or labels use release/effective timestamps, not file dates or period end dates.
  • Imputation does not backfill from future observations.

Red flags:

  • shift(-1) or negative periods in feature code.
  • .rolling(...) or .expanding(...) used on a feature without a following lag/shift.
  • bfill, fillna(method="bfill"), interpolation across the train/test boundary.
  • merge_asof(..., direction="forward").
  • Features named like future_*, next_*, target_*, label_*, outcome_*, realized_*, exit_*, or pnl_*.

Fitted Transform Scope

Every fitted object must be fit inside the training fold/window:

  • Scalers and normalizers.
  • Imputers.
  • Encoders.
  • PCA/UMAP/SVD.
  • Feature selectors.
  • Target encoders and count/frequency encoders.
  • Oversampling/SMOTE/class balancing.
  • Calibrators.
  • Threshold selectors.

Red flags:

  • fit_transform on the full dataset before split.
  • Global means/stds/min/max/quantiles used to normalize the full timeline.
  • Calibration trained on test predictions.

Label And Outcome Construction

Labels may use future data, but only inside label-generation artifacts that are excluded from model features. Check:

  • Label columns are explicitly excluded by name or schema.
  • Ambiguous/incomplete horizons are handled consistently.
  • Rows near the dataset end are not scored when labels are unavailable unless running true live-mode inference.
  • The target horizon cannot overlap the training window for the next fold without embargo.

Backtest And Simulation Logic

Check:

  • Decision prices, fills, spreads, borrow availability, liquidity, and latency are available at decision time or modeled realistically.
  • Position sizing uses only pre-trade account state.
  • Trade selection thresholds are fixed from training/validation, not final test performance.
  • Metrics include costs/slippage when the claimed result depends on tradability.

Comparable Experiment Checks

Fail a comparison when any of these differ unless the difference is the explicit treatment:

  • Dataset version/hash/row count.
  • Date range, cohort, universe, or entity membership.
  • Target definition or horizon.
  • Starting capital, budget, leverage, exposure, or cost model.
  • Split policy, embargo, retrain cadence, and lookback.
  • Metric definition, net/gross treatment, and aggregation level.

Review Standard

Lead with concrete findings and file/line references. For each issue, include:

  • What leaks or is logically faulty.
  • Why it changes evaluation results.
  • The minimal fix or guard.
  • Whether the result should be rerun.

If no leak is found, say which checks were performed and name the remaining unverified assumptions.

Common Fix Patterns

  • Replace random splits with chronological splits or blocked/time-series CV.
  • Fit scalers, imputers, encoders, feature selectors, calibration models, and threshold selectors inside each training window only.
  • Add an embargo/gap when labels or feature windows can overlap the scoring horizon.
  • Lag rolling/expanding features so the current prediction row cannot see its own outcome.
  • Use backward/as-of joins with explicit timestamp tolerance for point-in-time data.
  • Exclude labels, outcomes, future returns, post-event status fields, and model diagnostics from feature columns.
  • Separate validation/calibration/tuning windows from final test windows.
  • Save run metadata and fail comparisons when run controls differ.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.