Aejpol robustness

Robustness — Defending the Policy Estimate (aejpol-robustness)

When to trigger

• The headline causal estimate moves across specifications, or you do not yet know if it does
• A referee will ask "is this robust?" and you have no organized answer
• Inference (clustering, few clusters, multiple outcomes) is not yet airtight
• You need to show the policy conclusion, not just a coefficient, survives stress

Principle: robustness defends the policy conclusion, not the coefficient

At AEJ: Policy, robustness is judged by whether the policy takeaway is stable — if the headline estimate is the cost-per-job or the MVPF, show that number is stable, with its uncertainty, not merely that a regression coefficient stays significant. Organize the robustness program around the threats that would change the policy conclusion, and report enough that a skeptical referee can see each threat addressed.

Robustness by threat (each maps to a concrete check)

Threat to the policy conclusion	Check
Functional form / controls drive the result	Specification ladder; show the estimate across a coherent set, not a single lucky spec
Pre-trends / parallel-trends violation	Honest-DID (Rambachan–Roth) sensitivity bounds; placebo pre-period "effects"
Estimator bias under staggered timing	Re-estimate with ≥1 heterogeneity-robust DID estimator (CS / SA / BJS / dCDH)
Bandwidth / kernel (RDD)	Bandwidth sweep + bias-corrected CIs; donut-RDD if heaping at the cutoff
Weak / invalid instrument	Effective F; AR-robust CI; over-ID test if available
Wrong inference / few clusters	Wild-cluster bootstrap; report clustering level sensitivity
Multiple outcomes / specifications	Romano–Wolf / sharpened q-values; a specification curve where many specs are run
Confounding by an omitted policy/shock	Controls for co-timed policies; event-study around the focal reform only
Selection on unobservables	Oster (2019) δ / bounds; argue the implied selection is implausible
Sample composition / outliers	Drop influential jurisdictions; winsorize; alternative sample windows

Sensitivity that is policy-specific

• If the policy lesson depends on a welfare parameter you calibrate (discount rate, value of a statistic, recycling rule), report the lesson across a plausible range of that parameter, not one value.
• If external validity is the policy worry, show heterogeneity by jurisdiction characteristics and discuss which settings the estimate travels to.

Execution bridge (StatsPAI / Stata MCP)

Run the battery, don't just enumerate it. Full map: execution-with-mcp. AEJ: Policy evaluates programs and reforms; the design must carry a policy-relevant magnitude, not just statistical significance.

• Many outcomes / specifications: romano_wolf (step-down FWER, accounts for cross-test correlation) or benjamini_hochberg — report the adjusted threshold.
• OVB sensitivity: oster_delta / sensemakr — the confounder strength that would overturn the headline.
• Inference: wild_cluster_bootstrap (few clusters), twoway_cluster / conley.
• Re-fit off one handle: audit_result(result_id) lists the missing checks and the exact suggest_function for each — no guessing the battery.
• Exhibits: etable / did_summary_to_latex from the handle — no retyped numbers.

Keep the decisive checks in the body and the exhaustive (now actually-run) battery in the appendix. See the executed chain in the JF execution walkthrough.

Checklist

• The headline policy number (not just a coefficient) is shown stable across specs
• The single most likely referee threat is pre-empted with a dedicated exhibit
• At least one heterogeneity-robust estimator shown where staggered timing applies
• Inference stress-tested (wild-cluster / AR / multiple-testing as relevant)
• Selection-on-unobservables addressed (Oster bounds or equivalent)
• Calibrated welfare parameters varied across a defended range
• No "kitchen-sink" robustness with no narrative — each check answers a named threat

Anti-patterns

• A robustness section that is a wall of tables with no statement of which threat each rebuts
• Showing the coefficient is stable while the welfare/policy number is never re-derived
• A specification curve run but only the favorable region discussed
• Treating "still significant" as robustness while ignoring magnitude stability
• Calibrating one welfare parameter value and never probing it

Sequencing the robustness section for a referee

Order the section so a referee meets the answer before the doubt: (1) the main heterogeneity-robust estimate and its event-study; (2) the single most likely fatal threat with its dedicated check; (3) the inference stress-tests; (4) a compact specification curve or table of remaining variants; (5) the calibrated-parameter sensitivity for the welfare number. Each subsection ends with one sentence stating that the policy conclusion is unchanged, with its band — not merely that the coefficient stays signed.

Worked vignette (illustrative)

A staggered-DID estimate of a minimum-wage change on employment is the basis for a "small disemployment cost" policy claim. A referee will doubt staggered TWFE and pre-trends. The robustness program: CS and SA estimators (estimate within 10% of TWFE, illustrative), flat pre-period leads, an honest-DID bound showing the sign survives a pre-trend twice the largest observed lead, and wild-cluster inference across 30 states. The policy claim — disemployment cost per dollar of raised earnings — is re-derived under each and reported with its band.

Output format

【Headline policy number】the quantity whose stability you defend
【Top 3 threats】ranked by how badly each would change the conclusion
【Checks per threat】[threat → check → result]
【Inference】clustering / few-cluster / multiple-testing handling
【Calibrated-parameter sensitivity】range probed + conclusion stability
【Next step】aejpol-tables-figures