Backtest validation

Backtest Validation

A backtest is a claim, not a result. In-sample performance is the cheapest number in quant research: it rises with every parameter you tune and every variant you discard. This skill converts "the backtest looks great" into a defensible verdict — or an honest rejection.

The four ways a backtest lies

1. Selection bias. You tried N variants and reported the best. The maximum of N noise strategies has a positive expected Sharpe that grows with N. Ask the user how many variants were tried; disclose that number to the Deflated Sharpe Ratio. Undisclosed trials are the most common lie by omission.
2. Cost blindness. Gross returns ignore what turnover costs. An edge that dies at 10 bps per unit of turnover was never an edge — it was a liquidity donation you had not made yet.
3. In-sample memorization. Parameters fitted on the full history describe the past, not the future. A chronological in-sample / out-of-sample split is the minimum; walk-forward is better when the series is long enough.
4. Fragility. One draw of history is one draw. If a stationary block bootstrap says p(Sharpe ≤ 0) is 10%, the "edge" is statistically indistinguishable from luck regardless of how the equity curve looks.

Look-ahead and revision leakage are the fifth way — that is point-in-time-research's jurisdiction. When the input data has not passed PIT discipline, run that skill first; a gauntlet on contaminated data validates nothing.

Workflow

1. Establish what is being validated: returns series (frequency, gross or net), portfolio weights if available, benchmark, and — critically — how many variants were tried before this one. Record the answers.
2. If the returns come from a simulation the user or you just built, confirm the data joins are point-in-time clean (hand off to point-in-time-research if in doubt). Do not proceed on contaminated inputs.
3. Run the gauntlet: python scripts/validation_gauntlet.py --returns returns.csv [--weights weights.csv] --freq daily --trials N --json report.json With no data at hand, show the mechanics with --demo.
4. Read the verdict, not the equity curve. Report the honest metric set, the cost ladder, the IS/OOS split, the bootstrap CI, and the DSR together — never the Sharpe alone.
5. Translate flags into plain language for the user: what failed, why it matters, what would fix it (longer history, cost model, fewer variants, walk-forward). See references/methodology.md for the interpretation table.
6. Completion gate: claim the strategy "validated" only when the gauntlet exits 0 (PASS or WARN). On FAIL, the deliverable is the rejection and its reasons — that is a valid, useful research result. Label WARN verdicts as "conditionally validated" and enumerate the warnings.
7. Never extrapolate a validated backtest into a forward guarantee. The strongest permitted claim is: "survived the validation gauntlet on the disclosed information set".

Guardrails

• No conclusion from gross returns when turnover is knowable — run the ladder.
• No "best of N" reporting without disclosing N to the DSR.
• No deployment/paper-trading recommendation on a FAIL verdict.
• Missing inputs degrade honestly: the gauntlet emits no_cost_check / no_split_check warnings instead of silently skipping — surface them.
• Guaranteed-return language is prohibited regardless of verdict.