Falsification review
Use when a numerical claim, comparison claim, or headline result is about to leave `smokes/` or be written into `RESULTS.md`.From its SKILL.md
npx -y skills add atomicstrata/epistemic --skill falsification-reviewAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
- 12 stars12 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
22.0 KB, as published. Nobody here has run it
Related skills:
/skill:baseline-reproductionfor reproduced comparison targets,/skill:experiment-executionfor preregistered runs,/skill:surprise-triagewhen a reviewed claim still behaves strangely,/skill:verification-before-publicationbefore anything leaves the repo.
Falsification Review
Overview
This is the phase where the claim stops being yours and starts being evidence. Your job is not to help the claim survive; your job is to find the cheapest honest way to kill it before anyone else does.
A claim that only survives friendly reading is not a result. A claim that survives reproduced baselines, independent adversaries, and explicit disconfirming checks might be a result.
In this phase, you must:
- extract the exact claim from
RESULTS.mdor from the researcher's explicit statement - verify the experiment is anchored in
experiments/{id}/prereg.md - verify any cited baseline was actually reproduced at
experiments/repro_{name}/prereg.md - run
runFalsificationAdversary()fromsrc/adversary/dispatch.tsagainst the exact claim - require at least two adversary models different from the drafter model
- write one paper trail file per adversary to
experiments/{id}/falsifiers/{model}.md - decide
ALLOW,BLOCK, orALLOW WITH CAVEAT - keep provisional numbers in
experiments/{id}/smokes/until falsification passes - promote from
smokes/toRESULTS.mdonly after this phase clears the claim
The gates in this repo help, but they do not replace judgment. If a gate missed something, that does not grant permission.
The Iron Law
Your claim is guilty until proven defensible by ≥2 models
One model is not enough. Self-critique is not independence. Missing audit is not support.
When to Use
Use this skill when:
- you are about to move a number out of
experiments/{id}/smokes/ - you are drafting or editing
experiments/{id}/RESULTS.md - you are drafting or editing a repo-level
RESULTS.md - you are writing
beats,outperforms,matches,wins,better than, or any comparison claim - you want to treat a baseline as reproduced evidence instead of as a cited number
- the researcher asks,
Can we say this? - an experiment finished and now needs adversarial review before promotion
- a gate or reviewer says the claim is unsupported, caveated, or unreproduced
When NOT to Use
Do not use this skill when:
- you are still defining the claim; use
/skill:research-question - you are still locking the protocol; use
/skill:preregistration - you are still reproducing the comparison target; use
/skill:baseline-reproduction - you are still running the preregistered experiment; use
/skill:experiment-execution - the note will remain exploratory inside
experiments/{id}/smokes/ - the claim is already blocked because preregistration, judge lock, or execution integrity failed
- you are doing final publication packaging; use
/skill:verification-before-publication
Working Surface
These files and functions are the working surface for this phase. Read them. Use them. Do not guess.
| Surface | Why it matters |
|---|---|
HYPOTHESES.md | Registered claim, falsifier, baseline reference, judge reference |
experiments/{id}/prereg.md | Proof that the experiment existed before results |
experiments/{id}/judge.lock | Proof that the judge did not drift |
experiments/repro_{name}/prereg.md | Mandatory proof that the baseline was actually reproduced |
experiments/{id}/smokes/ | Provisional outputs only |
experiments/{id}/RESULTS.md | Confirmed experiment-level conclusions |
RESULTS.md | Optional repo-level mirror of confirmed conclusions |
experiments/{id}/falsifiers/{model}.md | Per-model falsification record |
.epistemic/cost-ledger.jsonl | Cost accounting for audit reruns |
OVERRIDES.md | Explicit exceptions with reasons |
src/state/repo.ts | Canonical state helpers and types |
src/adversary/dispatch.ts | Adversary dispatch and verdict parsing |
State helpers you will actually use here:
loadRepoState(cwd)for a quick snapshotloadHypotheses(cwd)andgetActiveHypothesis(entries)to recover the active hypothesisloadBaselines(cwd)andgetBaselineAgeDays(entry)to inspect baseline references and freshnessfileExists(path)to prove required artifacts existgetJudgeLock(cwd, id)andcomputeJudgeHash(judgeRef, id)to verify judge continuitygetHypothesisSpend(cwd, id)orgetAllHypothesisSpends(cwd)to inspect accumulated audit costappendCostRecord(cwd, record)if manual reruns created unlogged costupdateHypothesisStatus(cwd, id, status)when the claim is actually dead
Other helpers exist in src/state/repo.ts. Do not use saveHypotheses(...), hypothesisToMarkdown(...), or writeJudgeLock(...) here to paper over missing earlier work. This phase audits reality. It does not rewrite history.
src/adversary/dispatch.ts exposes runFalsificationAdversary({ claim, context, cwd }). Keep the hypothesis id in surrounding workflow even though the function itself does not currently take it; you still need it for file paths and status decisions.
The Process
-
Pin the exact claim
Start with the sentence that is asking for authority. Never falsify a vibe. Never falsify a softer paraphrase than the one about to ship.
Steps:
- Load
HYPOTHESES.mdwithloadHypotheses(cwd). - If the experiment id is not explicit, use
getActiveHypothesis(...). - Resolve the claim source in this order:
- the exact sentence already drafted in root
RESULTS.md - the exact sentence drafted in
experiments/{id}/RESULTS.md - the researcher's explicit statement in the current request
- the exact sentence already drafted in root
- If multiple versions exist, review the version that would actually ship.
- Quote the claim verbatim in your notes.
- Record the hypothesis id, metric, comparator, dataset, and destination file.
If the sentence does not say what improved, against what, on which task, and under which conditions, stop and rewrite it into a falsifiable claim before review.
- Load
-
Verify the experiment is real before reviewing the claim
Falsification review is not a substitute for missing protocol. If the experiment itself is unanchored, every later decision is fake confidence.
Steps:
- Check
experiments/{id}/prereg.mdwithfileExists(...). - If it is missing, BLOCK immediately.
- Use
loadRepoState(cwd)if you need a fast snapshot ofHYPOTHESES.md,BASELINES.md, and rootRESULTS.md. - Compare the drafted claim against the registered
HypothesisEntryfields fromsrc/state/repo.ts. - If the reviewed sentence silently changed metric, comparator, or scope from the registered claim, stop and send it back for correction.
A sentence can be numerically true on one metric and still invalid because it is no longer the preregistered claim.
- Check
-
Verify judge continuity instead of assuming it
Comparison claims are meaningless if the judge drifted halfway through the experiment. This repo has a judge-lock mechanism. Use it.
Steps:
- Read the expected judge from
HypothesisEntry.judgeRef. - Use
getJudgeLock(cwd, id)to check whetherexperiments/{id}/judge.lockexists. - If a lock exists, recompute the expected hash with
computeJudgeHash(judgeRef, id). - If the stored lock and computed hash do not match, BLOCK.
- If the lock is missing, stop and return the work to the judge-lock phase.
- Do not silently call
writeJudgeLock(...)here.
Late repair is not integrity. It is paperwork after the fact.
- Read the expected judge from
-
Prove the baseline was reproduced under your protocol
This is where comparison claims usually die. A cited baseline is not a reproduced baseline.
BASELINES.mdis context, not proof.Steps:
- Read
HypothesisEntry.baselineRefto identify the intended comparison target. - Use
loadBaselines(cwd)to locate the named baseline entry if one exists. - Normalize the baseline name into a stable
{name}token. - Check the proof path:
experiments/repro_{name}/prereg.md. - If
fileExists("experiments/repro_{name}/prereg.md")is false, BLOCK. - Use this exact blocking language:
Baseline not reproduced under your protocol. Missing experiments/repro_{name}/prereg.md. - If the file exists but is empty, vague, or clearly about another setup, treat it as missing.
- If a baseline entry exists in
BASELINES.md, usegetBaselineAgeDays(...)to check freshness. - If a reproduced baseline dossier contains a result file, read it before allowing any comparison sentence to survive.
The distinction is non-negotiable:
BASELINES.mdproves the number was recordedexperiments/repro_{name}/prereg.mdproves the baseline was reproduced under your protocol
If you cannot prove the second, you do not get to say
beats. - Read
-
Build the adversary packet correctly
runFalsificationAdversary()is only as good as the packet you send it. Friendly framing produces friendly nonsense.Steps:
- Use the real function signature from
src/adversary/dispatch.ts:runFalsificationAdversary({ claim, context, cwd }). - Keep the hypothesis id in surrounding workflow for paths and status updates.
- Build
contextfrom repo facts, not hype. - Include:
- the exact claim text
- the hypothesis id
experiments/{id}/prereg.md- the relevant evidence in
experiments/{id}/smokes/ experiments/repro_{name}/prereg.mdwhen the claim is comparative- the locked judge description from
HypothesisEntry.judgeRef - any caveat already known before adversarial review
- Determine the drafter model from
HypothesisEntry.judgeRefor from the active drafting context if it is more specific. - Require at least two adversary models that are different from the drafter model.
Each model must return the core payload:
{experiment, costEstimate, verdict, reasoning}. The canonical repo-side form isAdversaryVerdictinsrc/state/repo.ts:{ provider, model, name, experiment, costEstimate, verdict, reasoning }. - Use the real function signature from
-
Dispatch and distrust the first neat answer
The goal of the run is the strongest disconfirming experiment, not reassurance.
Steps:
- Call
runFalsificationAdversary({ claim, context, cwd }). - Confirm you received at least two model outputs.
- Confirm those outputs came from models different from the drafter model.
- Confirm each output has non-empty
experiment,costEstimate,verdict, andreasoning. - Exclude
cannot-auditfrom the set of passing audits. - If fewer than two independent audited verdicts remain after exclusions, BLOCK.
Read
src/adversary/dispatch.tscarefully before trusting the response shape blindly. The happy path returns canonical fields, but some fallback paths still emit legacy keys likemodelId,providerId, anddisconfirmingExperiment. If the response is malformed, normalize it before writing files and do not count the malformed response as a clean pass. - Call
-
Write the paper trail before deciding anything
Never make the verdict from scrollback memory. Write the evidence first.
Steps:
- Create or update
experiments/{id}/falsifiers/. - For each adversary verdict, write one file to
experiments/{id}/falsifiers/{model}.md. - Normalize model names into stable filenames.
- In each file include:
- claim text
- claim source path or researcher statement source
- hypothesis id
- drafter model
- adversary provider
- adversary model
- baseline reproduction path checked
- verdict
- proposed disconfirming experiment
- cost estimate
- whether that experiment has already been run
- evidence path proving run or non-run
- reasoning
- Determine
already been runfrom repo artifacts, not from confidence. - Compare the proposed disconfirming experiment against
experiments/{id}/smokes/,experiments/{id}/RESULTS.md, and existing falsifier files. - If manual reruns created unlogged cost, call
appendCostRecord(cwd, record). - If audit reruns are piling up, inspect
getHypothesisSpend(cwd, id)orgetAllHypothesisSpends(cwd).
Planned is not run. Mentioned is not run. Half-run is not run.
- Create or update
-
Apply the verdict matrix exactly
Once the evidence exists on disk, decide the outcome from the matrix below and nothing softer.
Condition Outcome Required action All independent audited verdicts are defensibleALLOWPromotion may proceed if no cheap unrun killer remains Any independent audited verdict is falsified-or-unreproducibleBLOCKStop promotion and record the blocking reasoning Mixed audited verdicts with at least one caveat-requiredand nonefalsified-or-unreproducibleALLOW WITH CAVEATTag the claim and ship the caveat with it Fewer than two independent audited verdicts remain after excluding cannot-auditBLOCKFix the adversary roster or configuration first Required baseline reproduction is missing BLOCKReproduce the baseline first Cheapest disconfirming experiment is <$1and unrunBLOCKRun the cheap killer before promoting Short form:
- all defensible ->
ALLOW - any falsified ->
BLOCK - mixed ->
ALLOW WITH CAVEAT - missing reproduced baseline ->
BLOCK <2independent audited models ->BLOCK- cheapest unrun disconfirming experiment
<$1->BLOCK
ALLOW WITH CAVEATis not decorative. Tag the claim. Attach the limitation where the claim appears. Do not bury it in another file.If the claim is actually dead, use
updateHypothesisStatus(cwd, id, "FALSIFIED"). - all defensible ->
-
Enforce the cheap-killer rule honestly
This is the part people try to skip because the number looks small. Do not help them do that.
Steps:
- Gather every proposed
experimentandcostEstimatefrom the adversary set. - Find the cheapest disconfirming experiment.
- Determine whether that exact experiment has already been run and documented.
- Count it as run only if the method matches and the evidence belongs to the current hypothesis.
- If the cheapest disconfirming experiment costs less than
$1and is unrun, BLOCK. - Use this exact blocking language:
Claim blocked pending cheapest disconfirming experiment (<$1) requested by falsification review.
If the strongest cheap killer is still sitting there unrun, the claim has not earned authority.
- Gather every proposed
-
Promote or refuse promotion cleanly
Promotion is the point where a provisional observation becomes repo memory.
Steps:
- If the outcome is
BLOCK, keep the evidence inexperiments/{id}/smokes/and keep the claim out of everyRESULTS.md. - If the outcome is
ALLOW, promote the reviewed result fromexperiments/{id}/smokes/toexperiments/{id}/RESULTS.md. - If the repo mirrors confirmed claims at the top level, update root
RESULTS.mdonly after the experiment-level file is correct. - If the outcome is
ALLOW WITH CAVEAT, promote only the reviewed sentence and attach the caveat directly beside it. - If the claim text changes after review, run falsification again.
- If the claim is blocked hard enough to end the hypothesis, route the work toward
/skill:kill-or-ship.
Promotion rule: only after falsification passes may anything move from smokes/ to RESULTS.md.
Laundering rule: promote now, rerun the falsifier later is not a workflow.
-
Close the loop before leaving the phase
A falsification review is not complete until the record is internally consistent.
Steps:
- Confirm the falsifier files exist under
experiments/{id}/falsifiers/. - Confirm the reviewed claim text on disk matches the text you actually reviewed.
- Confirm any caveat is attached wherever the claim appears.
- Confirm blocked claims stayed out of headline files.
- Confirm cost accounting is not silently missing.
- Confirm the hypothesis status still matches reality.
- If the claim survived but exposed weird instability, route the next step to
/skill:surprise-triage.
Common Rationalizations
| Excuse | Reality |
|---|---|
BASELINES.md already has the competitor number. | A cited number is not reproduced evidence. Proof is experiments/repro_{name}/prereg.md. |
| One strong model audit is enough. | The Iron Law requires ≥2 independent model reviews. |
| The drafter model challenging itself should count. | Self-critique is not independence. Same model does not count. |
| The draft sentence is basically the same as the reviewed one. | If the sentence changed, the claim changed. Re-review the shipped sentence. |
| The baseline paper is recent, so reproduction can wait. | Fresh citation is still not reproduction under your judge. |
cannot-audit is close enough to neutral. | Missing audit does not become supportive evidence. |
| The cheap killer is probably noise. | Then run it. If it costs <$1, guessing is more expensive than checking. |
| Mixed verdicts mostly mean yes. | Mixed means ALLOW WITH CAVEAT, not clean ALLOW. |
| We already spent enough on this hypothesis. | Then escalate to kill-or-ship. Do not lower the falsification bar. |
smokes/ looks stable, so promotion is safe. | Stable-looking provisional data is still provisional data. |
| We can promote now and clean up the caveat later. | If the caveat does not ship with the claim, the claim is misrepresented. |
| The function returned something shaped like a verdict, so it counts. | Malformed or legacy fallback output must be normalized and may still fail the audit minimum. |
Red Flags - STOP
Stop immediately if any of these are true:
- you cannot point to the exact sentence under review
experiments/{id}/prereg.mdis missingexperiments/{id}/judge.lockis missing or drifted- the claim is comparative and
experiments/repro_{name}/prereg.mdis missing - you are treating
BASELINES.mdas reproduction proof - fewer than two independent audited models remain after excluding the drafter model and
cannot-audit - one audited verdict is
falsified-or-unreproducible - the cheapest disconfirming experiment is
<$1and you cannot prove it was run - you are about to move anything from
smokes/intoRESULTS.mdbefore resolving the matrix - you are trying to paraphrase the claim after seeing adversary output
- you are calling
ALLOW WITH CAVEATa clean pass - you are about to silently repair missing historical artifacts instead of routing the work back to the failed phase
Any one of these means stop. Do not promote. Fix the broken premise first.
Good vs Bad
Claim extraction
Good
You load HYPOTHESES.md with loadHypotheses(cwd), recover hyp-017, and review the exact sentence in experiments/hyp-017/RESULTS.md: Model A beats Model B by 4.2 points on Dataset D under judge J. The adversary sees that exact sentence.
Bad
You summarize it as the model seems better on Dataset D before sending it for review. Now the falsifier is auditing a softer claim than the one that would ship.
Baseline proof
Good
You inspect HypothesisEntry.baselineRef, normalize the comparison target, and verify experiments/repro_llama-3/prereg.md exists and matches the task and judge.
Bad
You find ## Baseline: llama-3 in BASELINES.md and decide that is enough. It is not. That proves the number was recorded, not reproduced.
Adversary independence
Good You require two adversary models different from the drafter model and exclude any same-model self-audit from the pass count.
Bad You count the drafter model's self-critique as one of the two required adversaries. That is the same bias wearing a hat.
Cheap killer experiment
Good
Two adversaries propose disconfirming experiments costing $0.42 and $2.10. The $0.42 check has not been run, so you block with Claim blocked pending cheapest disconfirming experiment (<$1) requested by falsification review.
Bad
You say the $0.42 check is too minor to delay promotion. The cheaper the killer, the less excuse you have not to run it.
Mixed verdicts
Good
One adversary returns defensible; another returns caveat-required. No audited verdict says falsified-or-unreproducible, and no cheap unrun killer remains. You tag the claim ALLOW WITH CAVEAT and ship the caveat with the sentence.
Bad
You average the two judgments into a clean ALLOW. That is how caveats disappear and overclaiming starts.
Promotion discipline
Good
You keep provisional results inside experiments/{id}/smokes/ until the matrix resolves. Only then do you promote to experiments/{id}/RESULTS.md, and only then do you mirror anything into root RESULTS.md.
Bad
You copy a promising sentence into RESULTS.md first and promise to run falsification afterward. That is not sequencing. That is laundering provisional numbers.
Why This Matters
Most bad claims do not survive because the experiment runner is malicious. They survive because nobody forced the claim through hostile review before it became institutional memory.
This phase matters because it:
- prevents cited baselines from masquerading as reproduced baselines
- forces the cheapest honest killer to run before the claim earns authority
- keeps
smokes/provisional andRESULTS.mdmeaningful - turns caveated claims into explicitly caveated claims instead of quietly overstated ones
- leaves a durable audit trail in
experiments/{id}/falsifiers/ - gives you a clean cut between
ALLOW,BLOCK, andALLOW WITH CAVEAT
Once a sentence lands in RESULTS.md, people will reuse it. They will quote it in docs, planning, reviews, and future experiments. If the sentence is wrong, the error compounds.
That is why this phase is strict. If the claim cannot survive reproduced baselines, independent adversaries, and cheap disconfirming checks inside the repo, it has no business representing the repo outside it.
After this, use /skill:surprise-triage.