Reproducibility audit

Reproducibility Audit

Choose a concrete target output and attempt to regenerate it. Artifact availability alone is not reproduction.

Workflow

1. Identify one target table, figure, coefficient, metric, or file and its expected value or checksum.
2. Preserve artifact URLs, versions, release tags, commit identifiers, and local hashes.
3. Inventory raw data, derived data, preprocessing, analysis code, configuration, dependency pins, seeds, platform assumptions, and execution order.
4. Read references/good-enough-baseline.md and record every missing or ambiguous item.
5. Inspect entry points before execution. Run untrusted code in an isolated working copy with read-only source inputs, no credentials, least privilege, bounded resources, and no network unless the documented workflow requires a reviewed endpoint.
6. Start from the documented entry point in a fresh environment when feasible. Snapshot output paths before execution so pre-existing files cannot be mistaken for generated results.
7. Log exact commands, exit codes, standard output, standard error, elapsed time, and created or modified files.
8. Compare the generated target with the paper at the appropriate precision. Check values, sample sizes, labels, and transformations, not only whether a script exits successfully.
9. Repeat stochastic targets with documented seeds and assess expected variation.
10. Assign one status using references/reproduction-status.md.
11. Recommend the smallest changes that would make the target independently reproducible.

Rules

• Do not say reproduced unless the target was generated and compared.
• Distinguish missing materials, environment failure, undocumented execution, and output disagreement.
• Do not modify raw data or the only copy of an artifact.
• Do not silently repair code and then attribute the repaired result to the published package. Record patches separately.
• Treat unpinned dependencies, missing seeds, manual preprocessing, and unclear script order as findings.
• A zero exit code without a newly generated target is blocked, not reproduced or partially_reproduced. A before-and-after identity record is necessary but not sufficient: an unchanged pre-existing file is never evidence that the current run generated the target.
• Respect data licenses, access controls, confidentiality, and compute constraints.

Output

Return:

• target result and comparison tolerance;
• artifact identity and availability;
• environment and commands;
• generated output and comparison;
• missing or ambiguous ingredients;
• reproduction status;
• minimal corrective actions.