Baseline comparison review
Skill yeaight7/agent-powerups/plugins/machine-learning-ops/skills/baseline-comparison-review
Use when a new or more complex ML model is proposed and its value over simple baselines is not yet demonstrated -- before approving a new architecture or replacing an existing heuristic.From its SKILL.md
npx -y skills add yeaight7/agent-powerups --skill baseline-comparison-reviewAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 6 stars6 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
2.4 KB, 443 tokens by cl100k_base, as published. Nobody here has run it
Purpose
Machine learning models add technical debt. A complex model earns its place only by clearly outperforming a "dumb" baseline; this review makes that comparison explicit before a new model is approved.
When to Use
- A new model architecture is proposed for approval
- A complex model would replace an existing heuristic or rule
- Reported gains have not been compared against any baseline
Inputs
- The candidate model's evaluation results (metric + data split)
- The evaluation code, or enough dataset/task detail to define fair baselines
Workflow
- Define the naive baseline:
- Classification: predict the majority class.
- Regression: predict the mean or median of the training target.
- Time series: predict the last known value (naive persistence).
- Define the heuristic baseline: what simple if/else rule would a domain expert write?
- Evaluate both baselines on the same split and metric as the candidate model.
- Evaluate the delta: if the complex model only beats the heuristic baseline marginally (e.g., ~1%), recommend keeping the heuristic — the complexity is not worth the maintenance cost.
- Demand a baseline evaluation script before approving the new architecture, so the comparison is rerunnable.
Output
- A baseline-vs-model comparison on identical data and metric, with an explicit keep/replace recommendation that weighs maintenance cost
Verification
- Naive baseline defined and evaluated
- Heuristic baseline defined and evaluated (or explicitly ruled out with a reason)
- Candidate compared on the same split and metric as the baselines
- Delta judged against maintenance cost, not just statistical improvement
- A rerunnable baseline evaluation script exists
Failure Modes
- No heuristic baseline — comparing only against the naive baseline makes weak models look strong. Ask what rule a domain expert would write.
- Unequal comparison — baseline evaluated on a different split or metric than the model. Re-run both on identical data.
- Complexity bias — approving a model for a marginal gain without stating the maintenance cost in the recommendation.
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.
Gives 0 of the 12 instructions most review quality skills give in 443 tokens
Counted across 1,048 of the 1,783 authors here whose files we hold, read 2026-08-07
- Ask questions one at a timein 81 of 1048, across 64 files
- Provide a recommended answer for each questionin 73 of 1048, across 50 files
- Explore the codebase instead of asking answerable questionsin 66 of 1048, across 42 files
- Resolve dependencies between decisions one-by-onein 42 of 1048, across 17 files
- Interview the user relentlessly about the planin 38 of 1048, across 13 files
- Order findings by severityin 31 of 1048
- Resolve each branch of the decision treein 27 of 1048, across 5 files
- Run a grilling sessionin 26 of 1048, across 5 files
- Update CONTEXT.md immediately when a term is resolvedin 26 of 1048, across 11 files
- Propose precise canonical terms for vague languagein 25 of 1048, across 7 files
- Create documentation files lazilyin 24 of 1048, across 5 files
- Assign severity to every findingin 24 of 1048
Said here and by no other author read
- define the naive baseline
- define the heuristic baseline
- evaluate both baselines on the same split and metric
- evaluate the delta against maintenance cost
- demand a baseline evaluation script
- recommend keeping the heuristic for marginal gains
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.