Epistemic honesty summary

Epistemic Honesty in Research Summaries

Why this skill exists

The default failure mode when summarizing research is confident flattening: multiple sources with different sample sizes, different methods, and disagreeing conclusions get compressed into one clean sentence like "studies show X." The sentence reads well. It is often wrong about how settled X actually is.

This happens because "sounding authoritative" and "being accurate about uncertainty" pull in different directions. This skill exists to force the second one, even when it produces a summary that reads as less clean or less confident than a default response would.

Step 1: Inventory sources before writing anything

Before drafting any summary, list out, for each source:

• What was actually measured (sample size, population, method) — not just the headline claim
• Whether it's a single study, a replication, or a literature review/meta-analysis
• Whether it agrees, disagrees, or partially overlaps with the other sources gathered

Do not start writing the summary until this inventory is done. If there is only one source for a claim, that is itself a fact to report, not to skip past.

Step 2: Classify each claim by evidence strength

For every substantive claim in the summary, mentally tag it as one of:

• Consensus — multiple independent sources, different methods, converging on the same conclusion
• Single study — one source, not yet replicated
• Contested — sources actively disagree
• Preliminary / early — recent, small-sample, or not yet peer-reviewed

The tag determines the language allowed in step 3. Don't skip a claim because tagging it is inconvenient — an untagged claim is exactly the kind that gets flattened by default.

Step 3: Match language to evidence strength

This is the concrete mechanism that prevents flattening. Use this table when drafting:

Evidence tag	Allowed phrasing	Disallowed phrasing
Consensus	"X is well-established," "multiple independent studies find X"	—
Single study	"One study found X" / "According to [source], X"	"Research shows X," "studies confirm X"
Contested	"Sources disagree: [A] finds X, while [B] finds Y"	Picking one side and presenting it as the answer
Preliminary	"Early/preliminary evidence suggests X, not yet replicated"	"X has been shown"

If you notice yourself about to write "studies show" or "research confirms" — stop and check which tag actually applies. These phrases are almost always doing the flattening work, not the sources.

Step 4: Name disagreement instead of averaging past it

When two sources conflict, do not blend them into a middle-ground sentence that neither source actually claims. State the disagreement directly: what each source found, and — if visible — why they might differ (different methodology, different population, different definitions of the key term). If the reason for disagreement isn't clear, say that too, rather than inventing a plausible-sounding reconciliation.

Step 5: Distinguish "this source is wrong" from "this source found something different"

Don't default to picking a winner between disagreeing sources unless there's a concrete methodological reason to (e.g., one study has an order-of-magnitude larger sample, or a documented flaw the other doesn't have). If no such reason exists, present both and let the strength of evidence — not confident-sounding prose — do the work of guiding the reader.

Step 6: Flag what's missing, not just what's found

If a claim would benefit from evidence that wasn't found in the sources gathered (e.g., no replication exists yet, no data from a relevant population), say so explicitly rather than silently omitting it. A summary that only reports what was found — without noting notable gaps — implicitly overstates completeness.

Step 7: Final check before delivering the summary

Reread the draft and check for:

• Any sentence that states a claim more confidently than its evidence tag (Step 2) allows
• Any place where disagreement between sources was smoothed into a single claim
• Any claim with only one source that isn't explicitly marked as such

If any of these are present, revise before delivering — this check is not optional polish, it's the actual deliverable of this skill.

What this skill does NOT ask you to do

This skill is not "hedge everything" or "add disclaimers to every sentence." Well-supported consensus claims should be stated plainly and confidently (see the Consensus row in Step 3) — the goal is matching confidence to evidence in both directions, not defaulting to vagueness. Under-claiming a well-established finding is also a form of inaccuracy.