Alterlab ssci inference gate

Inference Gate — The Claim May Not Exceed the Design

Skill type: DISCIPLINE-ENFORCING. This is the terminal gate of the social-science methods spine. It reads the accumulated Design Passport and audits every inferential sentence against what actually identifies it — the design, the sample, and the uncertainty. It does not compute; it refuses conclusions the study cannot support.

The Core Rule

A CLAIM MAY NOT EXCEED ITS DESIGN, ITS SAMPLE, OR ITS UNCERTAINTY.
AUDIT EACH SENTENCE AGAINST WHAT ACTUALLY IDENTIFIES IT.

Three ceilings bound every claim. Design: causal language is licensed only if the design's identifying assumption (pinned by alterlab-ssci-design-gate) is named and defended — otherwise the claim is associational. Sample: a population generalization is licensed only by a probability sample of a frame that covers the target population (alterlab-ssci-sampling-gate). Uncertainty: a result is reported with an effect size and an interval, interpreted correctly — not as a bare significant/not-significant verdict. A sentence that breaks any ceiling is rewritten down to what the study supports.

When to Use This Skill

• "Help me write / check the results or discussion section."
• "My p-value is 0.03 — what can I conclude?" / "p was 0.20, so there's no effect, right?"
• "Does this finding mean X causes Y?"
• "Can I say this holds for [the broader population]?"
• "I ran 30 comparisons and three were significant — what can I report?"

Does NOT Trigger

The request is really about…	Route to	Why not this skill
Which statistical test to run	alterlab-test-selection-guard	Test choice, upstream of interpretation.
Executing the analysis / computing the estimate	alterlab-statistical-analysis	Computation, not claim audit.
Choosing the design & identifying assumption	alterlab-ssci-design-gate	Design routing happens first.
Reliability/validity of the measure	alterlab-ssci-measurement-gate	Measurement quality, a different ceiling.
Prose polish of an already-sound claim	alterlab-paper-writer	Writing, once the claim is audited clean.

The three ceilings

1. Design ceiling — causal language

Cross-check the claim's verb against design_type + identifying_assumption in the Passport:

• Causal verbs (causes, increases, reduces, improves, leads to, the effect of) require a design that identifies a causal effect and a defended assumption. Experiment → yes. QED → yes only if its assumption (parallel trends / exclusion / continuity / no time-varying confounders) is defended. Observational without a defended assumption → downgrade to associational (is associated with, predicts, correlates with).
• Never let a significant coefficient in an observational regression become "X causes Y."

2. Sample ceiling — generalization

• Statistical generalization to a population requires a probability sample of a covering frame (generalization: statistical in the Passport). Otherwise scope the claim to the sample ("among the surveyed students…") and offer only analytical generalization.
• Watch universal quantifiers ("adults", "people generally") attached to a convenience sample.

3. Uncertainty ceiling — p-values, CIs, effect sizes

Misreading	Correction
"p = 0.03 means a 3% chance the null is true."	p is P(data this extreme | null), not P(null | data).
"p > 0.05, so there is no effect."	Absence of evidence ≠ evidence of absence. Report the CI; a wide CI around zero is inconclusive, not a null proof.
"The 95% CI means a 95% probability the true value is in it."	The parameter is fixed; 95% is the long-run coverage of the procedure.
"It's statistically significant, so it matters."	Significance ≠ importance. Report the effect size and its interval; judge practical significance against it.
"Three of my 30 tests were significant."	Uncorrected multiplicity. Pre-specify, or correct (FDR / Bonferroni); disclose the family.
"I found the hypothesis the data supported."	HARKing / optional stopping. Separate pre-registered confirmatory from exploratory claims.

A stdlib linter that flags causal verbs, over-broad generalization, and p-value misuse against the Passport facts: scripts/claim_audit.py. Fuller treatment (ASA statement on p-values, New Statistics, multiplicity): references/inference_audit.md.

Reading the Design Passport

The gate consumes what upstream gates wrote and audits against it:

# from design-gate
design_type: observational
identifying_assumption: "conditional ignorability — NOT defended (no plausible ignorability argument)"
claim_type: associational
# from sampling-gate
sampling_method: convenience
generalization: analytical

Given this Passport, "remote work increases satisfaction among adults" fails two ceilings and is rewritten to "remote work is associated with higher satisfaction among the surveyed employees."

Self-Check Before Publishing

• Does every causal verb trace to a design + a defended identifying assumption?
• Is each generalization scoped to what the sample/frame supports (statistical vs analytical)?
• Are results reported as effect size + interval, with p-values interpreted correctly?
• Is a non-significant result framed as inconclusive (CI shown), not as proof of no effect?
• Are multiplicity and any exploratory/optional-stopping caveats disclosed?

References

• references/inference_audit.md — p-value/CI interpretation, multiplicity correction, the three-ceiling audit worked through.
• scripts/claim_audit.py — stdlib linter flagging causal-verb, generalization, and p-value overreach against Passport facts.

Part of the AlterLab Academic Skills suite.