Reproducibility gap detection

reproducibility_gap_detection

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

Identify and document gaps between published computational workflows and their reproducible instantiations (code, data, checkpoints, dependencies) by comparing article descriptions, source code documentation, and available artifacts. This skill is essential for flagging missing resources, undocumented preprocessing steps, or version pinning issues that would prevent independent replication.

When to use

When curating a scientific workflow (especially deep learning or data-processing pipelines) and you need to verify that all steps described in the publication have corresponding available code, data downloads, trained model checkpoints, and explicit dependency specifications. Apply this skill when the article claims 'checkpoints will be downloaded' or references external datasets but those resources are not yet located or their access paths are unclear.

When NOT to use

• Input is a theoretical method paper with no code or data artifacts — this skill is designed for computational/ML workflows with expected reproducible code and data.
• The article is already accompanied by a complete, validated Docker image or container with all dependencies pinned — use this skill only if you are verifying the container itself.
• You have not yet located the source repository or README — first retrieve those artifacts before applying this skill.

Inputs

• Published article text describing computational workflow, datasets, and tools
• Repository README and source code files (e.g., setup.py, requirements.txt, scripts/)
• URLs and download paths cited in the article or README
• Model checkpoint references and their expected locations

Outputs

• Gap report: list of missing or undocumented artifacts (checkpoints, datasets, preprocessing scripts)
• Reproducibility checklist: structured comparison of claimed vs. available resources
• Dependency resolution table: explicit versions and installation instructions needed

How to apply

Compare the article's workflow steps, dataset requirements, and tool dependencies against (1) the repository README and source code organization, (2) explicitly stated download locations (URLs, GitHub releases, cloud drives), (3) version constraints in setup files (requirements.txt, pyproject.toml, environment.yml), and (4) the completeness of preprocessing scripts. Document each gap by recording the missing artifact (e.g., checkpoint file, dataset preprocessing code, or dependency version), the location where it was claimed or expected, and whether it is retrievable or absent. Cross-reference filter steps and preprocessing parameters mentioned in the article with corresponding code implementations; if a filter (e.g., 'atom number ≤300') is described but no corresponding code enforces it, flag this as a gap.

Related tools

• molnetpack (PyPI package providing the high-level MolNet API; version and availability affect reproducibility of predictions) — https://pypi.org/project/molnetpack/
• 3DMolMS (Core neural network model; reproducibility depends on checkpoint availability and version alignment) — https://github.com/JosieHong/3DMolMS
• PyTorch (Deep learning framework; must be installed separately and version affects model checkpoint compatibility)
• RDKit (Chemistry toolkit for 3D conformer generation (ETKDGv3); required dependency with potential version sensitivity)

Examples

import torch
from molnetpack import MolNet
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
molnet_engine = MolNet(device, seed=42)
molnet_engine.load_data('./examples/demo_input.csv')
pred_df = molnet_engine.pred_msms(path_to_results='./output_msms.mgf', instrument='qtof')

Evaluation signals

• All datasets referenced in the article (HMDB, RefMet, QM9, METLIN, AllCCS, MoNA, NIST20) have explicit download instructions or scripts in the repository.
• Model checkpoints mentioned in the workflow (e.g., molnet_qtof_etkdgv3.pt) are locatable via a documented URL or GitHub release; their download is not blocked.
• Every preprocessing and prediction step in the article (e.g., 'python scripts/hmdb2pkl.py', 'scripts/predict.py --task msms') has a corresponding, executable script in the repository.
• Dependency versions are pinned or constrained in at least one of: setup.py, pyproject.toml, requirements.txt, environment.yml, or README; PyTorch installation instructions explicitly address OS/hardware variants.
• Filter criteria mentioned in the article (atom number ≤300, atom types, precursor types) are implemented in the preprocessing code; deviations from stated criteria are documented.

Limitations

• License restrictions (CC BY-NC-SA 4.0) on 3DMolMS may limit reproducibility in commercial or non-compliant contexts; datasets like NIST20 require separate academic licenses.
• 3D conformer generation using ETKDGv3 depends on RDKit version; older versions may produce different conformations, affecting model predictions reproducibility.
• Checkpoints downloaded from external sources (Google Drive, GitHub releases) may become unavailable or change without warning; reproducibility fragility depends on archive stability.
• The article describes preprocessing with multiple large datasets (HMDB, NIST20) but does not specify hardware/memory requirements; reproducing all steps may fail on resource-constrained systems.

Evidence

• [other] In the first running, the checkpoint (./checkpoint/molnet_qtof_etkdgv3.pt) will be downloaded from Google Drive.: "the checkpoint will be downloaded from [Google Drive]"
• [intro] 3D Molecular Network for Mass Spectra Prediction (3DMolMS) is a deep neural network model to predict the MS/MS spectra of compounds from their 3D conformations.: "deep neural network model to predict the MS/MS spectra of compounds from their 3D conformations"
• [methods] 3DMolMS has the following dependencies: Python 3.8+, PyTorch, RDKit, NumPy, Pandas, matplotlib, PyYAML: "dependencies: Python 3.8+, PyTorch, RDKit, NumPy, Pandas, matplotlib, PyYAML"
• [methods] python scripts/hmdb2pkl.py --data_config_path ./molnetpack/config/preprocess_etkdgv3.yml: "python scripts/hmdb2pkl.py --data_config_path ./molnetpack/config/preprocess_etkdgv3.yml"
• [methods] Download the HMDB molecules dataset from HMDB Downloads. The expected data directory structure is: |- data |- hmdb |- structures.sdf: "Download the HMDB molecules dataset from HMDB Downloads"
• [methods] Atom number ≤300; Atom types: 'C', 'O', 'N', 'H', 'P', 'S', 'F', 'Cl', 'B', 'Br', 'I', 'Na'; Precursor types: '[M+H]+', '[M-H]-', '[M+H-H2O]+', '[M+Na]+': "Atom number ≤300; Atom types: 'C', 'O', 'N', 'H', 'P', 'S', 'F', 'Cl', 'B', 'Br', 'I', 'Na'"
• [readme] PyTorch must be installed separately — see the official PyTorch site for the right command for your system.: "PyTorch must be installed separately — see the [official PyTorch site]"