Statistical model audit

Statistical Model Audit

Assess whether the model, uncertainty calculation, and diagnostics support the reported claim. Do not treat a familiar estimator or a passed diagnostic as proof that assumptions hold.

Workflow

1. Write the target quantity before reading coefficient signs: outcome, population, unit, time horizon, conditioning set, scale, and whether the goal is description, prediction, association, or causation.
2. Reconstruct the analysis data structure: observation unit, repeated measures, clusters, weights, censoring or truncation, sampling stages, missingness, and effective sample for each model.
3. Match the outcome support and data-generating structure to the likelihood, link, loss, or estimating equation. Record why the chosen family is appropriate and which alternatives remain plausible.
4. Reconstruct all preprocessing that can change the estimand or uncertainty: exclusions, transformations, winsorization, feature selection, imputation, lag construction, aggregation, and tuning.
5. State each material assumption and classify it as design-supported, empirically probed, externally justified, or untested. Use references/model-families.md for family-specific checks.
6. Verify uncertainty and tests: dependence structure, variance estimator, degrees of freedom, finite-sample correction, multiplicity, prior sensitivity, model selection, and repeated use of the data. Use references/inference-and-diagnostics.md.
7. Inspect diagnostics on the scale relevant to the claim. A residual plot, goodness-of-fit statistic, pre-test, or convergence message can reveal a problem but cannot establish every identifying or distributional assumption.
8. Run specification and sensitivity checks that target the most consequential assumption. Prefer a small number of pre-justified alternatives over an unstructured specification search.
9. Compare the conclusion with the model output on its actual scale. Check nonlinear transformations, interactions, marginal effects, subgroup claims, extrapolation, predictive calibration, and uncertainty propagation.
10. Report the smallest supported conclusion, unresolved assumptions, and the result that changes under a defensible alternative.

Routing

• Use $statistical-consistency-audit for deterministic arithmetic, intervals, reported test statistics, and rounding.
• Use $table-data-consistency-audit to trace model rows to generated artifacts, code, variables, and samples.
• Use $causal-claims-audit when the claim depends on treatment assignment or a named causal design.
• Use $argument-logic-audit when the numerical model may be correct but the premise-to-conclusion step is unsupported.
• Use $reproducibility-audit only when the target can be executed with the required artifacts. If invoked by $social-science-paper-audit, return the model assessment without calling the orchestrator again.

Output

For each model, record:

• claim, estimand or predictive target, and interpretation scale;
• data structure, model family, specification, and analysis sample;
• material assumptions and how each was assessed;
• uncertainty method, diagnostics, and sensitivity checks actually run;
• discrepancies between tables, code, and narrative;
• supported conclusion, impact, alternatives, and evidence needed to resolve uncertainty.

Classify a deterministic displayed contradiction as verified_reporting_error, a model or inference failure as identification_or_method_issue, an unsupported reasoning step as logic_or_argument_issue, and a signal requiring unavailable information as unresolved_anomaly. None of these classifications establishes intent.

Stop conditions

• Do not infer causation from model adjustment alone.
• Do not infer equivalence from a non-significant test.
• Do not interpret a coefficient without checking its link, transformation, interaction, and reference category.
• Do not call a model robust because one diagnostic or one alternative specification passed.
• Do not call an analysis reproduced unless a named output was regenerated and compared.