Table data consistency audit

Table and Data Consistency Audit

Build a cell-addressed audit trail from the printed table back to the analysis sample. Treat rendered source pages as authoritative for what was published; use extracted text only for navigation and search.

Required workflow

1. Preserve and identify every artifact inspected: paper, supplement, appendix, data release, code revision, log, exported table, and correction. Record versions, hashes or stable identifiers, and access dates when available.

2. Render every relevant PDF page. Visually verify headers, row labels, panels, decimal points, minus signs, superscripts, footnotes, significance stars, and sample-size notes before transcribing a value.

3. Assign every audited value an exact address:

   • artifact and version;
   • PDF page plus table, panel, and section;
   • complete row path and column path, including spanning headers;
   • displayed string, unit or scale, precision, stars, and attached notes;
   • screenshot or rendered-page reference for high-impact cells.

   If the source supplies too little information for a physical address, create a provisional semantic address with a stable render ID and explicitly list the missing page, panel, header, or coordinate fields. Do not call a provisional address complete.

4. Trace each high-impact cell through this provenance chain:

   printed cell
     -> exported result, table object, log, or intermediate file
     -> generating command, function, model, and configuration
     -> analysis variable, transformation, weight, and estimator
     -> raw variable or source field
     -> inclusion, exclusion, missingness, and final analysis sample
   

   Mark every edge as verified, documented_only, inferred, missing, or not_checked. Use missing only for an expected artifact or link that is unavailable in the stated scope; use not_checked when an artifact was available but not inspected, or when an upstream missing artifact prevented examination of downstream edges. Qualify missing as not_supplied, not_found_after_search, or confirmed_absent_from_named_package. Never bridge a missing edge by assumption.

5. Reconstruct the sample flow. Reconcile recruitment or source rows, eligibility, exclusions, deduplication, missing outcomes or covariates, attrition, subgroup filters, panel observations, and the N used by each table column. Check whether a complete-case or model-specific sample changes silently across columns.

6. Audit data transformations and joins. Check key uniqueness, join cardinality, duplicate creation or loss, aggregation level, denominator, weights, units, currencies, price bases, dates, time zones, signs, percentage versus proportion scales, logs, standardization, winsorization, and category coding.

7. Reconcile presentation metadata. Confirm that table notes, model labels, reference groups, fixed effects, clustering, weights, adjusted tests, significance thresholds, and stars match the generating output and code. Treat a correct coefficient with a wrong label, unit, N, or footnote as a separate reporting problem.

8. Recompute deterministic identities in batches when the repository checker supports them. Load $statistical-consistency-audit, inspect the current supported kinds, and encode one stable case ID per claim. A batch file may contain a JSON array:

   python -m paper_audit list-checks
   python -m paper_audit check table-audit-cases.json
   

   Record input_decimals only for continuous values that are displayed after rounding. Omit it for exact discrete counts, integer sample sizes, and exact category totals; assigning decimal-place uncertainty to an exact count can create a false rounding_feasible result. Use exactly one reported-output precision rule and retain the command, output, and exit code. Do not force an unsupported relationship into the nearest check kind. Treat rounding_feasible as unresolved and an input error as a failed check setup, not a paper error.

   The checker schema is intentionally closed and does not carry source-cell addresses, footnote scope, or semantic premises. Keep those fields in the cell-address and provenance matrices, link them to the stable case ID, and do not insert unsupported metadata into checker JSON. Translate a footnote into a numeric identity only after its applicable columns, definitions, and scope are verified; otherwise retain it as a manual metadata check.

9. Validate the table/data sidecar independently of checker cases before relying on its links:

   python scripts/validate_table_record.py table-audit-record.json
   

   The standalone wrapper runs from an isolated installation of this skill. In a checkout of the full toolkit, the equivalent command is python -m paper_audit validate-table-record table-audit-record.json. Use schemas/table-audit-record.schema.json for editor or CI validation and the runtime command for cross-record references, provenance dependencies, sample-flow identities, and join row-count constraints. Exit code 0 means the sidecar is internally coherent, 1 means its structure or declared relationships are invalid, and 2 means the input file could not be read or parsed. A valid sidecar does not establish that a transcription is accurate, a PDF was parsed automatically, or an analysis was reproduced.

10. Confirm every substantive mismatch independently. Reinspect the rendered page, rule out OCR or transcription error, propagate feasible rounding intervals, and consider documented weighting, covariance, degrees of freedom, multiplicity, and version differences.

High-risk manual checks

Give extra scrutiny to structures that are hard to verify mechanically:

• merged or multilevel headers, multi-panel tables, continuation pages, transposed layouts, and repeated row labels;
• OCR-sensitive minus signs, decimal separators, superscripts, subscripts, inequalities, and blank versus zero cells;
• stars based on adjusted, one-sided, clustered, bootstrap, or small-sample inference;
• totals built from rounded components, weighted subgroup aggregates, changing denominators, and suppressed categories;
• transformed coefficients, marginal effects, standardized estimates, exponentiated results, and values copied from another model;
• generated LaTeX, spreadsheet links, manual edits after export, stale caches, and a table whose code path differs from the documented command;
• joins or reshapes whose row count is plausible but whose entity, time, or aggregation grain changed.

Sampling a few cells can support a scoped spot check only. Do not generalize a clean sample to the entire table. For a central table, prioritize all Ns, primary estimates, uncertainty measures, derived totals, and cells feeding abstract or conclusion claims.

Decision boundaries

• Call a displayed contradiction verified only when the exact source cell is visually confirmed and no feasible rounding, note, adjustment, or version explanation reconciles it.
• Call a provenance edge missing only for the identified materials. Distinguish not supplied, not found after a documented search, and confirmed absent from a named public package.
• Do not say a table was reproduced unless the reviewed code and required data generated the target output and it was compared with the publication.
• Do not infer fabrication, falsification, selective intent, or misconduct from a mismatch, broken join, duplicate, missing artifact, or unusually clean table.
• A passed arithmetic check establishes only that relationship. It does not validate the source data, estimator, construct, or conclusion.

Output

Return:

1. scope and artifact inventory;
2. a cell-address matrix with exact locations and transcription evidence;
3. a provenance matrix showing every chain edge and status;
4. sample-flow, join, unit, note, star, and rounding checks;
5. batch-check commands and machine-readable results;
6. verified contradictions, unresolved discrepancies, and provenance gaps kept separate;
7. checks that passed, limits of coverage, local and paper-level impact, and the evidence needed to resolve each uncertainty.

Use references/record-spec.md to keep cell addresses, provenance edges, deterministic case IDs, and manual metadata premises linked without adding unsupported fields to checker JSON.