agentsclimarketplace

Backtest comparator

Skill baronguyen001/ai-automation-skills/skills/backtest-comparator

8 production-tested Claude skills: automation, Gemini cost/structured output, OSS bounty scouting, ML validation.

Install
npx -y skills add baronguyen001/ai-automation-skills --skill backtest-comparator

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Compare strategy variants across multiple years or folds and flag the ones that only look good on average - rank by mean fold score but surface dispersion and worst-fold so a variant that overfits one lucky period does not win silently. Use for compare backtest variants, flag overfit strategy, per-fold consistency, year by year backtest, or which parameter set is robust.

SKILL.md

2.8 KB, as published. Nobody here has run it

Backtest Comparator

Use this skill when several strategy variants or parameter sets each have per-year (or per-fold) results and you need to pick the robust one, not the one with the highest average. A variant can win on the mean while quietly losing on an individual fold; this comparator ranks by mean but also reports standard deviation and the worst fold, then flags overfit and unstable variants so they can't slip through.

When to invoke

  • User says: "compare these backtest variants" / "which parameter set is robust" / "flag the overfit one" / "year-by-year results"
  • Code in the conversation uses: a sweep that produced per-fold or per-year scores for multiple variants.

When NOT to invoke

  • You only have a single aggregate number per variant (no per-fold breakdown to judge consistency).
  • The task is to build the validation split itself (use [[walk-forward-runner]] first, then compare its folds here).

Concrete example

User input:

Three variants, four years of returns each. Tell me which is actually robust vs which just got lucky one year.

Output:

variant              mean      std    worst  verdict
------------------------------------------------------
cross_only          0.188    0.027    0.150  robust
macd_filter         0.095    0.011    0.080  robust
pullback_ema21      0.075    0.205   -0.120  OVERFIT (positive mean, loses on a fold)

Code:

# Copy assets/compare.py into your project, then:
from compare import compare_variants, format_table

folds = {
    "cross_only":     [0.18, 0.22, 0.15, 0.20],
    "macd_filter":    [0.10, 0.09, 0.11, 0.08],
    "pullback_ema21": [0.40, -0.12, 0.05, -0.03],
}
print(format_table(compare_variants(folds)))

Pattern to apply

  1. Collect per-fold scores per variant in chronological order (one number per year/fold).
  2. Compute mean, standard deviation, worst fold, and best fold for each variant.
  3. Rank by mean, but never decide on mean alone.
  4. Flag OVERFIT when the mean is positive yet a fold is negative; flag UNSTABLE when dispersion exceeds the mean.
  5. Prefer the variant whose worst fold still holds up over the one with the flashiest average.

Reference: assets/compare.py.

Source

Distilled from production use across the author's automation projects. v1.0.0. See also: [[walk-forward-runner]], [[config-audit-checklist]].

→ Build the full runnable bot with Trawlkit.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.