Evidence linking

evidence_linking

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

Link predicted MS/MS spectra and molecular property predictions (RT, CCS) back to their 3D conformational inputs and training data provenance, enabling validation of model outputs against known standards (HMDB, MoNA, NIST20) and traceability of predictions through the neural network pipeline.

When to use

After generating MS/MS spectra, retention time, or collision cross section predictions using 3DMolMS, you need to validate predictions against reference spectra (MoNA, NIST20, HMDB), verify input molecule conformations were preprocessed correctly (ETKDGv3), or establish reproducibility by linking outputs back to the specific checkpoint, dataset partition, and filter criteria (atom count ≤300, supported atom types, precursor type) used in training or inference.

When NOT to use

• Input molecules contain unsupported atom types (not in {C, O, N, H, P, S, F, Cl, B, Br, I, Na}) or exceed 300 atoms — these are automatically filtered and will not be present in output.
• Ground-truth spectra are unavailable or from a different ionization mode (QTOF vs. Orbitrap) than the model checkpoint — comparison will not be valid.
• Predictions are being made without a valid checkpoint file or the checkpoint is incompatible with the conformation preprocessing method used (conformation method mismatch will cause silent or hard failures).

Inputs

• preprocessed molecule dataset (PKL format) with 3D conformations (ETKDGv3 method)
• ground-truth MS/MS spectra (SDF or MGF files from MoNA, NIST20, or HMDB)
• checkpoint file (PyTorch .pt model, e.g., molnet_qtof_etkdgv3.pt)
• metadata about training/test split, atom type filters, precursor type filters

Outputs

• evaluation results DataFrame (CSV) with similarity scores per molecule
• similarity histogram (PNG) showing distribution of cosine similarity values
• predicted spectra in MGF format (linked to input molecule IDs)
• predicted RT or CCS values in CSV format (linked to input molecule IDs)
• trace log linking each prediction to input conformation, checkpoint, and filter criteria

How to apply

Load the test data (PKL format) containing preprocessed 3D conformations, run inference via molnet_engine.pred_msms(), pred_rt(), or pred_ccs() to generate predictions in MGF or CSV format, then use molnet_engine.evaluate() to compute similarity metrics (e.g., cosine similarity) against ground-truth spectra extracted from MoNA, NIST20, or HMDB. Record the checkpoint version (e.g., molnet_qtof_etkdgv3.pt), conformation method (ETKDGv3), atom/precursor type filters applied, and data split used. Use the evaluation results DataFrame and plots (eval_similarity_hist.png) to confirm predictions meet quality thresholds and trace any failures back to unsupported input filtering or atom type/precursor type mismatches.

Related tools

• molnetpack (Python API for loading data, running inference (pred_msms, pred_rt, pred_ccs), training, and evaluation; core interface for evidence linking) — https://pypi.org/project/molnetpack/
• 3DMolMS (Deep neural network model that learns molecular representations from 3D conformations; generates predictions that must be linked back to input conformations and training checkpoints) — https://github.com/JosieHong/3DMolMS
• RDKit (Generates and validates 3D molecular conformations (ETKDGv3) used as inputs; dependency for preprocessing pipeline)
• MoNA (Reference library of experimental MS/MS spectra (LC-MS/MS QTOF and Orbitrap formats) used as ground truth for evaluation)
• NIST20 (Reference library of high-resolution MS/MS spectra (hr_msms_nist.SDF) for validation under academic license)
• HMDB (Reference molecular structure and spectra database; used for dataset preprocessing and ground-truth comparison)
• PyTorch (Deep learning framework; manages checkpoint loading, model inference, and gradient computation for training)

Examples

results_df = molnet_engine.evaluate(test_pkl="./data/qtof_etkdgv3_test.pkl", pred_mgf="./output_msms.mgf", result_path="./eval_results.csv", plot_path="./eval_similarity_hist.png")

Evaluation signals

• Cosine similarity between predicted and ground-truth spectra ≥ 0.7 for ≥80% of molecules in test set (indicates successful MS/MS prediction transfer)
• All predicted molecules are traceable to input conformations with metadata (mol_id, checkpoint version, conformation method, filters applied) recorded without gaps
• Histogram of cosine similarities shows unimodal distribution centered ≥0.6 (bimodal or left-skewed distribution indicates systematic failure, e.g., wrong checkpoint or preprocessing mismatch)
• Predictions for molecules filtered during preprocessing (atom count >300, unsupported atom types, unsupported precursor types) are absent from output (confirms filter enforcement)
• Ground-truth spectra from input dataset match the instrument mode of the checkpoint (QTOF vs. Orbitrap mismatch detected by MGF header or manual inspection)

Limitations

• Atom count must be ≤300; molecules exceeding this threshold are automatically filtered and not evaluated.
• Only supported atom types (C, O, N, H, P, S, F, Cl, B, Br, I, Na) and precursor types ([M+H]+, [M-H]−, [M+H-H2O]+, [M+Na]+) can be processed; others are silently excluded.
• Conformation method must match the checkpoint (e.g., ETKDGv3); mismatches can degrade prediction quality without explicit error message.
• Ground-truth spectra must be available in compatible format (SDF, MGF, PKL); absence of reference data prevents evaluation.
• Model evaluation uses cosine similarity, which may not capture all aspects of spectral quality (e.g., intensity prediction accuracy); complementary metrics (e.g., spectral entropy) may be needed for thorough validation.

Evidence

• [readme] This model's molecular representation, learned through MS/MS prediction tasks, can be further applied to enhance performance in other molecular-related tasks, such as predicting retention times (RT): "This model's molecular representation, learned through MS/MS prediction tasks, can be further applied to enhance performance in other molecular-related tasks, such as predicting retention times (RT)"
• [methods] 3DMolMS can be used to generate a reference library of small molecule MS/MS spectra: "3DMolMS can be used to generate a reference library of small molecule MS/MS spectra"
• [readme] The evaluate() method computes similarity metrics against ground-truth spectra: "Evaluate against ground truth: results_df = molnet_engine.evaluate(test_pkl="./data/qtof_etkdgv3_test.pkl", pred_mgf="./output_msms.mgf", result_path="./eval_results.csv","
• [methods] Supported precursor types for evaluation: "Precursor types: '[M+H]+', '[M-H]-', '[M+H-H2O]+', '[M+Na]+'"
• [methods] Reference libraries used for validation: "MoNA-export-All_LC-MS-MS_QTOF.sdf, MoNA-export-All_LC-MS-MS_Orbitrap.sdf"
• [readme] Checkpoint automatic download and versioning: "Predict and save to MGF (checkpoints are downloaded automatically)"
• [methods] Conformation preprocessing method specification: "Preprocess HMDB dataset using ETKDGv3 conformation: python scripts/hmdb2pkl.py --data_config_path ./molnetpack/config/preprocess_etkdgv3.yml"