Script reproducibility verification

script-reproducibility-verification

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

A systematic workflow for executing deposited computational scripts to regenerate published simulation outputs, validating that reported Type I error rates and statistical power results can be independently reproduced. This skill ensures methodological transparency and computational reliability by comparing regenerated summary statistics against originally reported values.

When to use

When you have access to a repository containing simulation scripts (e.g., Type I error and power simulation code) and need to verify that the published findings—specifically reported error rates or power estimates—can be independently regenerated by re-running the deposited code. Use this skill when reproducibility verification is a project requirement or when you suspect discrepancies between documented and executable outputs.

When NOT to use

• Scripts are not available or repository is not accessible; script reproducibility verification cannot proceed without executable code.
• Reported values are not clearly documented in the article or supplementary materials; validation requires a concrete reference target.
• The goal is exploratory analysis or extension of the method rather than verification of existing reported results.

Inputs

• GitHub repository URL (e.g., github.com/CharlieCarpenter/PaIRKAT)
• Deposited simulation scripts (Type I error and power simulation code)
• Published reported values (Type I error rates, power estimates, summary statistics)
• Simulation scenario definitions (sample sizes, effect sizes, test parameters)

Outputs

• Regenerated Type I error summary statistics
• Regenerated power summary statistics
• Validation report comparing regenerated vs. reported values
• Reproducibility assessment (pass/fail or discrepancy log)

How to apply

Clone or obtain the source repository from the provided GitHub URL. Locate and load the simulation scripts (e.g., SimulationFunctions) containing the Type I error and power simulation code. Execute the Type I error simulation script to compute error rates and generate summary statistics for each tested scenario. Execute the power simulation script to compute statistical power across all tested conditions and generate corresponding summary statistics. Collect all generated outputs and systematically validate them against the reported values in the article or supplementary materials, checking for numerical agreement within expected floating-point tolerance. Document any discrepancies, including script version, execution environment (R version, package versions), and parameter configurations.

Related tools

• R (Execution environment for simulation scripts and statistical computation)
• PaIRKAT SimulationFunctions (Source scripts containing Type I error and power simulation code to be executed and validated) — github.com/CharlieCarpenter/PaIRKAT

Examples

# Clone repository, load scripts, execute Type I error simulation
git clone github.com/CharlieCarpenter/PaIRKAT && cd PaIRKAT && Rscript SimulationFunctions/TypeI_simulation.R && Rscript SimulationFunctions/Power_simulation.R

Evaluation signals

• Regenerated Type I error rates match reported values within numerical tolerance (typically ±0.001–0.01 depending on simulation sample size and reported precision).
• Regenerated power estimates match reported power values across all tested scenarios (effect sizes, sample sizes, alpha levels).
• Summary statistics (e.g., mean, SD, CI bounds) for both error and power outputs agree with published tables or figures.
• Script execution completes without errors and produces output files with expected structure and format.
• Environment metadata (R version, package versions, random seed handling) is documented and consistent between runs.

Limitations

• Floating-point arithmetic and random number generation may introduce small numerical differences; exact bit-for-bit reproduction is typically not achievable across different hardware or software versions.
• Script reproducibility does not verify the correctness of the underlying statistical method—only that reported values can be regenerated from the code.
• If the repository lacks version tags or environment specifications (e.g., sessionInfo, renv/conda lockfiles), reproducing results with identical package versions may be difficult.
• No changelog was found in the repository, so it is unclear whether scripts have been modified since publication or whether historical versions are available.

Evidence

• [other] Can the Type I error and power simulation outputs from the PaIRKAT repository be successfully regenerated by executing the deposited simulation scripts?: "Can the Type I error and power simulation outputs from the PaIRKAT repository be successfully regenerated by executing the deposited simulation scripts?"
• [other] Execute the Type I error simulation script to compute error rates and generate summary statistics.: "Execute the Type I error simulation script to compute error rates and generate summary statistics."
• [other] Execute the power simulation script to compute statistical power across tested scenarios and generate summary statistics.: "Execute the power simulation script to compute statistical power across tested scenarios and generate summary statistics."
• [other] Collect and validate all generated simulation summary outputs against reported values.: "Collect and validate all generated simulation summary outputs against reported values."
• [readme] TypeI and Power simulation scripts: "TypeI and Power simulation scripts"