Social science paper audit

Social Science Paper Audit

Build a traceable correction record. Do not infer intent from an anomaly.

Required workflow

1. Preserve the supplied paper, supplement, code, and data as read-only source artifacts.
2. Record title, authors, venue, DOI, version date, and every artifact URL. Distinguish “not found” from “does not exist.”
3. Extract both layout-aware text and rendered pages. Use rendered pages to verify tables, equations, footnotes, minus signs, superscripts, and OCR-sensitive symbols.
4. Inventory the paper’s central claims before searching for anomalies. Map each claim to its premises, table, figure, equation, data source, estimator, stated identifying assumptions, and conclusion scope.
5. Route checks as follows:
   • Use $table-data-consistency-audit to trace claims through displayed cells, generated outputs, analysis code, sample construction, and source variables. Prioritize this route for dense tables and data-supported claims that are costly to verify manually.
   • Use $statistical-consistency-audit for arithmetic, intervals, test statistics, sample sizes, and formula identities.
   • Use $statistical-model-audit for model-family choice, dependence, missing data, diagnostics, uncertainty, and model-dependent interpretation outside a named causal design.
   • Use $figure-visual-consistency-audit when quantitative evidence is encoded in plot geometry, including scatterplots, coefficient plots, event studies, binned plots, fitted lines, or figure-to-data inconsistencies that text extraction cannot preserve.
   • Use $argument-logic-audit for contradictions and unsupported steps between premises, evidence, interpretations, and conclusions.
   • Use $causal-claims-audit for causal estimands, research design, identifying assumptions, and inference.
   • Use $reproducibility-audit when executable code, required data, and configuration are jointly available, including in a self-contained archive. Inventory partial materials without pretending that the target can be executed.
   • Use $openalex plus authoritative journal or repository pages for metadata, references, corrections, expressions of concern, and retractions.
   • Use $research-integrity-audit to classify the combined evidence and control allegation language.
6. Independently recompute every high-impact numerical finding when the reported inputs or required artifacts permit it. Log exact inputs, commands, outputs, rounding rules, and assumptions; otherwise record the unperformed check and missing ingredient as a scope limitation.
7. Search for benign explanations before upgrading a finding: displayed rounding, one-tailed tests, multiple-testing adjustments, clustering, weights, missing observations, extraction errors, or an undisclosed version difference.
8. State whether each issue changes one displayed value, one result, an identification claim, or the paper’s central conclusion.
9. Validate the structured report before publication:

python -m paper_audit validate-report audit-report.json

Evidence classes

Use exactly these classes:

• verified_reporting_error: a deterministic contradiction survives rounding and documented alternatives.
  • identification_or_method_issue: the design, estimator, or inference does not support the stated claim.
  • logic_or_argument_issue: the stated premises or cited evidence do not support the conclusion, or claims contradict one another, independently of whether the estimator was implemented correctly.
• reproducibility_gap: a required artifact or executable step is confirmed missing from, or fails in, the identified public package. Materials merely not supplied to the current auditor are a scope limitation until publication availability is checked.
• unresolved_anomaly: a signal needs source data, better extraction, or author clarification.
• official_finding: an identified journal, institution, or formal investigation published the finding.

Only official_finding may be described as officially established misconduct. An automated flag, unusual pattern, or verified arithmetic error is not by itself evidence of intent.

Minimum finding record

For every finding, retain:

• exact PDF page and table, figure, section, or equation;
• verbatim reported values or a screenshot reference;
• the claim-to-table-to-code-to-data support chain, including any missing link;
• independent method and result;
• rounding interval, sample definition, degrees of freedom, weighting, and variance assumptions;
• plausible alternative explanations;
• local and paper-level impact;
• verification status and evidence class.

Read references/audit-protocol.md for the domain checklist, references/tool-routing.md before choosing external tools, and references/report-template.md when drafting the final report.

Stop conditions

• Do not say “reproduced” unless the target output was generated and compared.
• Do not call two estimates equivalent because a difference test was not significant.
• Do not treat a list, proxy, model prediction, or imputed value as ground truth without independent validation.
• Do not treat a logically valid argument as empirically true, or a statistically sound estimate as support for a conclusion outside its stated population, time, construct, or estimand.
• Do not publish a misconduct accusation from this workflow. Preserve evidence and request clarification or formal review.