Research critic
Skill jajupmochi/agent-harness/skills/general/research-critic
Use AT EVERY RESEARCH STEP (hypothesis design, experiment setup, result interpretation, conclusion writing) to challenge soundness BEFORE committing the step. Catches confirmation bias, p-hacking, leakage, weak baselines, ungrounded claims, ad-hoc thresholds, survivorship bias. Apply automatically whenever about to write a paper claim, design an ablation, interpret a number, or update a results doc. Pairs with code-verifier (which audits artifact authenticity).From its SKILL.md
npx -y skills add jajupmochi/agent-harness --skill research-criticAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
7.8 KB, ~1.8k tokens by cl100k_base, as published. Nobody here has run it
/research-critic
Adversarial review of the inferential chain BEFORE a claim is published.
Master TOC
- Core principle
- When to invoke
- The Six-Question Audit
- Anti-bias checklist
- Output format when critic flags an issue
- Quick check (single line)
- Anti-rationalisation
- Companion
Core principle
A real experiment can still be interpreted wrongly. Numbers don't speak for themselves — they require a chain of inference from {data, method, comparison} to {claim}. Each link is a place to lie to yourself. This skill challenges each link adversarially before the claim is published.
When to invoke
Before ANY of:
- Writing a result line in a paper draft / RESULTS.md
- Closing an experiment ticket as "done"
- Adding a number to a table or figure
- Saying "our method beats X" or "our method matches Y"
- Concluding from an ablation
- Committing a "headline" milestone in git
- Sharing results with a collaborator / advisor
The Six-Question Audit
Run all six. Each NO answer is a STOP — fix before claiming.
Q1. Is the hypothesis falsifiable?
| Sufficient | Insufficient |
|---|---|
| "Method X lifts metric Y by ≥ Z pp over baseline on dataset D, N folds" | "Method X improves Y" (improves how? on what?) |
| "Coverage matches target 1−α within Δ across α∈{...}" | "Calibration works" |
Red flags: vague verbs (improves, outperforms, works); no specific magnitude; no specific dataset / metric / comparator.
Q2. Does the experiment design test the hypothesis?
Check that:
- Independent variable changes ONLY the thing under test
- All other knobs (seed, encoder checkpoint, pool, split, threshold, normalisation) are matched between conditions
- Dependent variable measures what the claim says (mAP vs recall@1 vs precision are different)
- Sample size sufficient for the claimed effect size
Common breaks:
- "Method A: 60%, Method B: 65%" but A used 4 folds and B used 1 fold → not comparable
- "Cross-domain transfer X%" but no in-domain baseline at SAME pool size → ambiguous denominator
- "Our method works at scale" tested only on 200 samples
- "Multi-seed × narrow-TTA helps" but no anti-control (variants may anti-compound)
Q3. Is the comparison fair?
| Fair | Unfair |
|---|---|
| Same pool, same metric, same code path, only the named knob differs | Old baseline number from a different setup |
| Both numbers from runs in this commit | One from prior paper, one from new code |
| Statistical test acknowledges variance | "X > Y by 0.1 pp" claimed as a win with σ ≈ 1 pp |
Anti-pattern: "free improvement" trap. If method B looks better than A "for free", check whether B was tuned (knob search) while A was the first config tried. Random initialization can give 1-2 pp swings; tuning B but not A is unfair.
Q4. Could the result be explained by leakage / artifact?
Audit:
- Training set ∩ Test set must be empty (string identity ok; semantic identity not enough — e.g. same writer in train and test still leaks)
- Validation set used to tune hyperparams must be held out from test-set metrics
- Per-class evaluation: are test classes ever seen during training? (Closed-vocab vs open-vocab)
- For ranking metrics: is the ranking model trained on the same query-relevance signal it's evaluated on?
- For cross-domain transfer: truly never exposed to eval domain? (Including via pretrained weights.)
Q5. Is the conclusion proportional to the evidence?
| Conclusion | Evidence required |
|---|---|
| ✅ "Method X +A pp over Y (N-fold paired test, p<0.05)" | N-fold mean+std, paired test |
| ⚠ "X is the best alignment method" | Tested vs Chamfer & TPS; not vs ICP, RANSAC, all alternatives |
| ❌ "X solves task T" | Single benchmark only |
Rules of thumb:
- 1 dataset → "we observe on X"
- 2-3 datasets → "we observe across X, Y, Z"
- 4+ datasets with diverse properties → "generalises" (still hedge: under conditions Z)
- Never: "solves", "best", "universal" without proof of impossibility for alternatives
Q6. Does the result survive plausible alternative explanations?
For each headline claim, list at least TWO alternative explanations and rule them out:
Claim: "Cross-domain transfer is 79.2%"
Alt 1: "Maybe pretrained encoder is doing all the work, not the alignment module"
Rule out: Run with raw baseline → if comparable, alignment adds nothing
Status: ✅ raw is 12%, with alignment 13.3% → adds 1.3pp
Alt 2: "Maybe transfer ratio is high because in-domain reference is also poor"
Rule out: Check absolute in-domain numbers
Status: ⚠ in-domain is 16.8% → ratio high partly because BOTH are low; reframe
Alt 3: "Maybe noise — single training run"
Rule out: Multi-seed
Status: ❌ only 1 seed. Caveat the claim or rerun.
If ANY alt is not ruled out, the claim is over-stated.
Anti-bias checklist
Applies to every result line:
- Pre-registered prediction? (or at least: designed before seeing data?)
- All conditions disclosed, not just winners
- Effect size given, not just "significant"
- Variance / std reported
- Random seed disclosure (multi-seed if practical)
- Baseline strength documented (not a strawman)
- Code + data available for replication (commit hash, data version)
- Ablation isolates each contributing component
Output format when critic flags an issue
[RESEARCH CRITIC — FLAG]
Question: Q<N>. <question summary>
Claim under audit:
"<exact wording from draft / table / commit>"
Issue:
<which assumption is violated or unverified>
Evidence needed to support claim:
- <experiment / number / control>
- <experiment / number / control>
Conservative restatement (if claim cannot be fully supported):
"<weaker but defensible claim>"
Action:
[ ] Run additional experiment: <command>
[ ] OR weaken claim and proceed
Quick check (single line)
Before pasting any number into a paper / RESULTS.md / commit:
"Can I defend this number against an adversarial reviewer who knows the field, the data, and my code? If not — what's missing?"
If the answer involves wishful thinking ("they probably won't ask", "reviewers usually accept this", "it's standard"), STOP. Fix or weaken.
Anti-rationalisation
| Excuse | Reality |
|---|---|
| "It's the trend that matters" | Then claim the trend, not the number |
| "Reviewers won't notice" | They will |
| "Standard in the field" | Standard ≠ correct |
| "Just for this draft" | Drafts ossify into papers |
| "It's only 0.1 pp" | Then say "no difference" |
| "Pool size doesn't matter much" | Then test it |
| "Run another seed later" | Run it now or hedge now |
Companion
code-verifier— audits artifact authenticity (Layer 1-3). Use BEFORE this skill (genuine artifact, then check inferential chain).always-on-verificationrule (in this lib) — when to invoke both.- The
superpowers:requesting-code-reviewskill complements with external review of the code path producing the numbers.
Provenance
Originally authored as a user-level always-on gate; consolidated into agent-harness for cross-project reuse. Pairs with code-verifier for full claim-defensibility coverage.
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.
Gives 0 of the 12 instructions most docs writing skills give in ~1.8k tokens
Counted across 1,951 of the 3,904 authors here whose files we hold, read 2026-09-06
- Use third-person for skill descriptionsin 54 of 1951, across 35 files
- Start descriptions with Use whenin 43 of 1951, across 29 files
- Run baseline scenarios before writing any skillin 40 of 1951, across 26 files
- Use active voicein 40 of 1951, across 36 files
- Map file responsibilities before defining tasksin 36 of 1951, across 29 files
- Use checkbox syntax for tracking stepsin 35 of 1951, across 27 files
- Ask one question at a timein 35 of 1951
- Offer execution options after saving the planin 33 of 1951, across 24 files
- Include complete code in every stepin 33 of 1951, across 27 files
- Design units with clear boundaries and interfacesin 31 of 1951, across 23 files
- Announce the skill usage at the startin 30 of 1951
- Verify agent compliance after adding the skillin 29 of 1951, across 17 files
Said here and by no other author read
- run the six-question audit before publishing any claim
- list two alternative explanations for every headline claim
- rule out alternative explanations before finalizing a claim
- disclose all conditions instead of only winners
- report effect size and variance for all results
- weaken claims that cannot be fully supported by evidence
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.