Hype check

Hype Check

Most misleading claims aren't false — they're true about the wrong thing. "77% of voters encountered a deepfake" is true and tells you almost nothing about whether deepfakes changed a single vote (independent fact-checkers found under 1% of flagged election misinformation was actually AI-generated, and a large field experiment found AI microtargeting no more persuasive than generic messaging). The claim measured prevalence and let you infer impact. This skill pulls those two apart and asks the three questions hype is built to skip: what's the denominator, where's the counterfactual, and what else could explain this?

The core split: prevalence is not impact

Every big claim quietly conflates two different measurements. Name which one the claim actually supports:

• Prevalence / existence — "X is happening," "X exists," "N% encountered X," "incidents of X rose." Usually the easy thing to measure, and usually all the evidence actually shows.
• Measured causal impact — "X caused Y," "X changed the outcome by magnitude Z." The hard thing, requiring a counterfactual, and usually absent.

A claim that establishes prevalence and lets you infer impact is the single most common move in hype. When you see a scary/exciting stat, first ask: does this measure that the thing happens, or that it matters? They are not the same evidence, and the second is the one you actually care about.

The three questions hype skips

1. Where's the denominator?

Big raw numbers are meaningless without the base they're drawn from.

• "5 documented cases of X preceding harm" — out of how many total users? A numerator with no denominator can't be a rate, and can't tell you if X is rare or common.
• "Incidents up 1,400%" — from what base? A jump from 2 to 30 is a huge percentage and a tiny absolute. Always convert to rate-per-exposed and compare to a baseline.
• Missing-denominator claims are often unknowable rates dressed as alarming ones. Say when the denominator genuinely doesn't exist — that's a finding.

2. Where's the counterfactual?

Impact is a comparison to what would have happened otherwise. If there's no comparison, there's no impact claim.

• "Users of X had outcome Y" — compared to non-users? Matched how? Without a control, Y might have happened anyway.
• "Sales rose after we shipped X" — versus a world without X, or just versus last quarter (when sales were rising regardless)? Two numbers moving requires no causal story.

3. What's the confound?

Before crediting X, list what else changed at the same time.

• Correlated trends: news layoffs rose and AI content farms grew — shared upstream drivers (ad economics), not necessarily one causing the other. Treat as confounded, not isolated.
• Reverse causation: "AI-leader firms outperform" — or do well-run firms both adopt AI and outperform for unrelated reasons?
• Selection: who's in the sample, and did they self-select in a way that produces the result?

The forecasting-hype variant

A special case: "N% of jobs/tasks at risk" style projections. Exposure is not outcome. A famous 2013 forecast put 47% of US jobs at high automation risk within a decade; unemployment then fell to multi-decade lows, and the occupations rated highest-risk sometimes grew. Task-level exposure measures technical susceptibility, not job loss — the outcome depends on bargaining power, retraining, and demand. When you see "X% exposed," refuse to read it as "X% will be lost." (For the deeper version, see the protect-the-skill and analogy-check skills.)

Procedure

1. State the claim and what it feels like it proves.
2. Classify: does the evidence establish prevalence, or measured impact? (Usually prevalence.)
3. Denominator: is there a base rate? If a raw count, convert to a rate; if none exists, say so.
4. Counterfactual: is there a comparison to "otherwise"? If not, the impact claim is unsupported.
5. Confounds: list what else could produce this (correlated trend, reverse causation, selection).
6. Restate what the claim actually supports — usually narrower than the headline — and what evidence would be needed to support the strong version.

Output template

CLAIM: <as stated> | FEELS LIKE IT PROVES: <the strong reading>

MEASURES: prevalence | measured impact  (which does the evidence actually show?)
DENOMINATOR: <base rate present? raw count → rate? or genuinely unknowable?>
COUNTERFACTUAL: <compared to what? present or absent?>
CONFOUNDS: <correlated trend | reverse causation | selection — what else fits?>

WHAT IT ACTUALLY SUPPORTS: <the honest, usually-narrower claim>
TO SUPPORT THE STRONG VERSION, YOU'D NEED: <the missing evidence>

Example (compressed)

Claim: "AI companions are causing the teen loneliness epidemic — 70% of teens have used one!"

MEASURES: prevalence. "70% have tried one" says usage is common. It says nothing
  about causing loneliness.
DENOMINATOR: the 70% is fine as a prevalence rate — but there's no rate linking
  use to onset of loneliness.
COUNTERFACTUAL: absent. No comparison of matched users vs. non-users over time.
  And teen sadness was already trending up for years before companions existed —
  adoption is a single recent point on a pre-existing curve.
CONFOUNDS: reverse causation is live and evidence-backed — lonelier teens seek
  companions, so use may be a *symptom*, not a cause. Correlational data even
  shows heavier use tracking more loneliness, which fits both directions.

WHAT IT ACTUALLY SUPPORTS: AI companion use is common among teens, and heavier
  use correlates with loneliness — direction unresolved.
TO SUPPORT "CAUSING THE EPIDEMIC": you'd need longitudinal/experimental evidence
  isolating companion use from pre-existing loneliness. It doesn't exist yet.

Gotchas

• Deflating hype is not denying the phenomenon. The thing can be real and worth watching while the causal/impact claim is unsupported. Say both.
• "Unknowable denominator" is a real answer. Some rates can't be computed (the behavior evades measurement). Report that instead of inventing a number.
• Cuts both ways — check exciting claims too. FOMO hype ("everyone's 10x now") gets the same three questions as fear hype.
• Pairs with validate-the-claim (source incentives), grade-the-evidence (tiers), and disaggregate (a real aggregate impact can still hide opposite effects by group).