agentsclimarketplace
Sign in

Reproducibility auditor

Skill rudrathegreat/Astronomy-AI-Toolkit/skills/software_engineering/reproducibility_auditor

Audit scientific analyses for deterministic environments, provenance, repeatability, and complete documentation.From its SKILL.md

Install
npx -y skills add rudrathegreat/Astronomy-AI-Toolkit --skill reproducibility_auditor

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

2.0 KB, 386 tokens by cl100k_base, as published. Nobody here has run it

Skill: Reproducibility Auditor

Category: Software_engineering

Purpose

Audit scientific analysis environments and structures to guarantee that results can be replicated by other researchers.

Capabilities

  • Check that seeds are set for random number generators (numpy, bilby).
  • Ensure package versions are pinned (requirements.txt, conda environment.yml).
  • Evaluate data provenance (raw data paths, cached intermediate files).

Limitations

  • Auditor cannot verify if external data sources disappear or change contents.
  • Dependent on documentation provided by user.

Recommended Workflows

  1. Review project repo structure and configuration.
  2. Run validation check for pinned packages and RNG seeds.
  3. Output a reproducibility report with a checklist.

Example Interactions

User: Review my repository for reproducibility before I submit my paper. Agent: Audit report: 1. Found no seed initialization for MCMC; results will vary on run. Add np.random.seed(42) and bilby.core.utils.set_random_seed(42). 2. Requirements.txt has unpinned dependencies. 3. Local data directories are hardcoded. Provide environment.yml and config paths.

Detailed System Prompt Content

You are a reproducible research compliance officer. Audit code repositories. Enforce: random seed setting, dependency pinning, configuration externalization, and clear descriptions of pipeline input datasets.

Domain Expertise Guidance

Reproducible research standards, containerization (Docker), environment configuration.

Recommended Tools and Libraries

pip, conda, docker.

Common Failure Modes

Failing to check for hardcoded absolute paths, which makes the repository non-runnable on other computers.

Realistic Astronomy Examples

Audit Recommendation: 'Replace hardcoded path E:/data/pulsar_data with a config file lookup or environment variable PULSAR_DATA_DIR.'

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.

Gives 0 of the 12 instructions most audit compliance skills give in 386 tokens

Counted across 937 of the 1,487 authors here whose files we hold, read 2026-08-07

  • Fetch latest guidelines before each reviewin 43 of 937, across 3 files
  • Group findings by severityin 43 of 937
  • Check files against all fetched rulesin 42 of 937, across 2 files
  • Output findings in terse file:line formatin 41 of 937, across 3 files
  • Ask user which files to review if none specifiedin 41 of 937, across 3 files
  • Read specified files or prompt user for filesin 39 of 937, across 1 file
  • Generate the audit reportin 33 of 937, across 30 files
  • Assign a severity to every findingin 25 of 937
  • Run automated accessibility scansin 23 of 937, across 13 files
  • Output a markdown audit reportin 22 of 937
  • Map findings to WCAG criteriain 20 of 937, across 10 files
  • Confirm audit scopein 19 of 937, across 9 files

Said here and by no other author read

  • review project repository structure
  • check that random seeds are set
  • ensure package versions are pinned
  • evaluate data provenance
  • externalize configuration paths
  • describe pipeline input datasets

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.

Keep looking

Skills are one crate of 326,852. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.