Grade the evidence

Grade the Evidence

Most bad analysis fails the same way: it states a shaky claim in the same confident voice as a rock-solid one. The reader can't tell which is which, so the whole thing gets trusted — or distrusted — as a block. This skill fixes that by forcing two separate judgments on every empirical claim: how strong is the evidence, and — independently — how confident should we be. High agreement among experts about thin data is not the same as strong data, and collapsing the two is how false certainty gets manufactured.

The core move: two axes, never one

Judge every load-bearing claim on both axes and report them separately:

1. Evidence strength — what kind of evidence backs this, on the tier scale below.
2. Confidence — how much the claim should move a decision, after accounting for the evidence tier, disagreement among credible sources, and the time horizon.

They come apart constantly. "Nearly every expert agrees, but no one has measured it" is high-agreement / low-evidence — report it that way, not as high confidence. Borrowed from how the IPCC keeps its evidence/agreement axis separate from its confidence axis, and from GRADE in medicine.

The six-tier evidence scale

Tag each empirical claim inline with the strongest tier it honestly earns:

1. RCT / quasi-experimental — randomized trials, well-identified natural experiments, diff-in-diff with credible identification. The gold standard; still fallible.
2. Observational-quantitative — correlational studies, large-N surveys, usage telemetry, economic indicators without causal identification. Real signal, no clean causation.
3. Expert-consensus — convergent judgment across independent experts absent decisive data. Strong on agreement, weak on measurement.
4. Theoretical / mechanistic — derived from theory or a formal model, untested at the relevant scale.
5. Analogical — reasoning from precedent; must state the points of disanalogy (see the analogy-check skill).
6. Speculative — coherent extrapolation with no direct support. Permitted only in explicit scenario/what-if framing, never as a load-bearing claim, and always visually marked as speculation.

Tiers 1–3 can anchor a confident claim. Tiers 4–6 cannot carry a verdict on their own.

The ⚠ Contested / Thin flag

Any claim resting on tier 4–6 evidence — or tier 1–3 evidence with material disagreement among credible sources — gets a ⚠ Contested/Thin flag plus one line saying which kind of problem it is:

• Data doesn't exist yet (unmeasured) → resolvable by waiting for studies.
• Data exists but is disputed (contested) → credible sources genuinely disagree.
• Definitionally hard to measure (e.g. "died well," "understands") → may never resolve cleanly.

The remedy differs for each, so naming the type is not decoration — it tells the reader whether to wait, to weigh both sides, or to stop expecting a number.

The language rule that makes it bite

A verdict built on majority thin evidence may not use confident directional language. Ban "AI will…", "this proves…", "X causes Y." Use hedged, mechanism-anchored constructions instead:

• "Early evidence suggests…" (tier 2, thin)
• "The mechanism implies but has not confirmed…" (tier 4)
• "Experts broadly expect, though no study has measured…" (tier 3)

If you catch yourself writing a confident sentence, check the tier of what it rests on. If the tier can't carry the confidence, the sentence is wrong — soften the claim or find stronger evidence.

Watch for confidence that outruns competence

A fluent, assertive register is not evidence. This is the single most common tell of a weak claim dressed up strong — in AI output and in human writing alike. Concretely: strong confidence markers ("clearly," "definitely," "obviously") correlate poorly with accuracy, and the dominant error is rarely a fabricated fact — it's interpretive overconfidence: stating an attributed opinion as settled fact, or generalizing a narrow finding into a broad one. When you see certainty markers, that's a prompt to check the tier underneath, not to relax.

Procedure

1. Extract every load-bearing empirical claim (the ones a decision actually rests on — skip the throat-clearing).
2. Tag each with its evidence tier (1–6) and a one-clause reason.
3. Flag any tier 4–6 or disputed tier 1–3 with ⚠ Contested/Thin + the problem type.
4. Set confidence per claim, explicitly noting where it's lower than the tier alone would suggest (disagreement, long horizon, small/unreplicated sample) — a single unreplicated study is tier 1 evidence but not yet a settled finding.
5. Rewrite any sentence whose confidence outruns its tier.
6. If asked for a bottom line, give it — but built from the graded claims, not a vibe.

Output template

CLAIM: <the specific empirical claim>
  Evidence: Tier <n> — <what kind, one clause>
  Flag: <none | ⚠ Contested/Thin — unmeasured|disputed|hard-to-measure>
  Confidence: <low|moderate|high> — <why, esp. if below the tier>
  Rewrite: <hedged version, if the original overclaimed>

Example

Input: "Studies show AI tutoring boosts learning outcomes 30%."

Graded:

CLAIM: AI tutoring raises learning outcomes ~30%.
  Evidence: Tier 1 for scaffolded systems (Harvard Kestin RCT, 0.73–1.3 SD);
            Tier 2/mixed for unscaffolded "just ask ChatGPT" use.
  Flag: ⚠ Contested/Thin — the "30%" collapses two different interventions.
        Scaffolded tutoring that withholds answers helps; unscaffolded
        answer-giving shows measured skill LOSS under proctored testing.
  Confidence: moderate, and only for scaffolded designs. Near-zero for the
              blanket "AI tutoring" version — the effect flips by design.
  Rewrite: "Scaffolded AI tutoring that withholds answers shows large RCT
           gains; unscaffolded answer-giving does not, and can reduce
           unaided performance."

The claim wasn't true or false. It was two claims wearing one number — and only grading the evidence separated them.

Gotchas

• Don't grade rhetoric, grade the load-bearing claims. Not every sentence needs a tier — only the ones a decision rests on.
• A single RCT is tier 1 but not settled. Unreplicated, small-sample, or single-site results get tier-1 strength but capped confidence. Say so.
• Vendor benchmarks are self-reported. They're often tier 2 at best and conflicted; route capability/performance claims through the validate-the-claim skill.
• "Experts agree" is tier 3, not tier 1. Agreement about unmeasured things is exactly the case this skill exists to catch.
• Pairs with disaggregate. A single graded claim can still hide opposite effects across groups — grade and disaggregate for anything consequential.