Output grounding

output_grounding

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

Validate and ground predicted MS/MS spectra, retention times, and collision cross sections against experimental reference data or ground truth by computing similarity metrics, spectral alignment scores, and error distributions. This skill ensures that neural network predictions are scientifically interpretable and fit-for-purpose before integration into metabolomics workflows.

When to use

After running 3DMolMS predictions (MS/MS spectra, RT, or CCS) on a test set with known experimental ground truth. Use this skill when you need to quantify model accuracy, detect systematic biases, or validate that predictions meet the similarity or error thresholds required for downstream compound annotation or library matching.

When NOT to use

• When ground truth experimental data is unavailable or the test set is not representative of the deployment domain (e.g., test set from only one instrument type but model will be applied to spectra from multiple instruments).
• When the goal is exploratory prediction without validation requirements—use prediction directly if no accuracy threshold is defined.
• When predictions are being used in a fully unsupervised setting (e.g., generating hypothetical library entries for novel compounds with no measured reference).

Inputs

• predicted MS/MS spectra in MGF format (from pred_msms())
• predicted retention times in CSV format (from pred_rt())
• predicted collision cross sections in CSV format (from pred_ccs())
• test dataset pickle file (.pkl) containing ground truth experimental values and molecular structures
• test compound metadata (SMILES, InChI, precursor m/z, instrument type)

Outputs

• evaluation results table (CSV) with per-compound similarity scores, errors, and pass/fail flags
• summary statistics (mean, median, std. dev. of similarity or error metrics)
• histogram of similarity or error distribution (PNG/PDF plot)
• scatter plot of predicted vs. experimental values (for RT, CCS)
• flagged outliers or low-confidence predictions for manual review

How to apply

Load the predicted outputs (MGF for spectra, CSV for RT/CCS) alongside the corresponding test pickle file containing ground truth experimental data. Compute similarity metrics (e.g., cosine similarity between predicted and experimental MS/MS spectra) and error distributions (e.g., mean absolute error for RT or CCS predictions). Generate summary statistics, histograms of similarity distributions, and optional scatter plots of predicted vs. observed values. Use these metrics to assess whether the model meets application requirements (e.g., cosine similarity > 0.7 for spectral matching in reference library construction) and to identify failure modes or outlier molecules that may require special handling.

Related tools

• MolNet.evaluate() (Core evaluation method that computes similarity metrics and error statistics between predicted and ground truth spectra/properties; generates plots and summary CSV) — https://github.com/JosieHong/3DMolMS
• molnetpack (Python package providing the MolNet engine and evaluate() method for grounding predictions) — https://pypi.org/project/molnetpack/
• matplotlib (Visualization library used to generate histograms and scatter plots of similarity/error distributions)
• Pandas (Data manipulation library for loading, aggregating, and exporting evaluation results)

Examples

molnet_engine.evaluate(test_pkl='./data/qtof_etkdgv3_test.pkl', pred_mgf='./output_msms.mgf', result_path='./eval_results.csv', plot_path='./eval_similarity_hist.png')

Evaluation signals

• Cosine similarity between predicted and experimental MS/MS spectra falls within expected range (e.g., median > 0.7 for high-quality QTOF predictions); low outliers indicate problematic molecules or model weaknesses.
• Mean absolute error (MAE) or root mean square error (RMSE) for RT and CCS predictions is within acceptable bounds for the analytical platform (typically <10% relative error for well-characterized compounds).
• Distribution of similarity/error metrics is approximately normal or unimodal with no large unexpected gaps, suggesting consistent model behavior across the test set.
• Visual inspection of predicted vs. experimental scatter plots shows points clustered near the y=x diagonal with no systematic bias (e.g., consistent underprediction at high m/z).
• Fraction of predictions passing a user-defined threshold (e.g., cosine similarity > 0.7) aligns with the model's reported validation accuracy from the paper or README.

Limitations

• Evaluation accuracy depends on the quality and representativeness of ground truth data; if the test set is biased toward certain compound classes or instrument conditions, evaluation metrics may not generalize to deployment scenarios.
• Supported atom types are restricted ('C', 'O', 'N', 'H', 'P', 'S', 'F', 'Cl', 'B', 'Br', 'I', 'Na'); compounds with other elements or >300 atoms are filtered out, so evaluation on such compounds is not possible.
• Precursor type coverage is limited to '[M+H]+', '[M-H]-', '[M+H-H2O]+', '[M+Na]+'; evaluation of adducts outside this set cannot be performed.
• Predictions are instrument-specific (QTOF vs. Orbitrap); evaluation results for one instrument type do not directly validate performance on another instrument.

Evidence

• [readme] Evaluate against ground truth: results_df = molnet_engine.evaluate(test_pkl='./data/qtof_etkdgv3_test.pkl', pred_mgf='./output_msms.mgf', result_path='./eval_results.csv', plot_path='./eval_similarity_hist.png',): "Evaluate against ground truth results_df = molnet_engine.evaluate( test_pkl="./data/qtof_etkdgv3_test.pkl", pred_mgf="./output_msms.mgf", result_path="./eval_results.csv","
• [other] Atom number ≤300, Atom types: 'C', 'O', 'N', 'H', 'P', 'S', 'F', 'Cl', 'B', 'Br', 'I', 'Na', Precursor types: '[M+H]+', '[M-H]-', '[M+H-H2O]+', '[M+Na]+': "Atom number ≤300 and Atom types: 'C', 'O', 'N', 'H', 'P', 'S', 'F', 'Cl', 'B', 'Br', 'I', 'Na' and Precursor types: '[M+H]+', '[M-H]-', '[M+H-H2O]+', '[M+Na]+'"
• [readme] 3DMolMS is a deep neural network model to predict the MS/MS spectra of compounds from their 3D conformations: "3DMolMS is a deep neural network model to predict the MS/MS spectra of compounds from their 3D conformations"
• [readme] This model's molecular representation, learned through MS/MS prediction tasks, can be further applied to enhance performance in other molecular-related tasks, such as predicting retention times (RT) and collision cross sections (CCS).: "This model's molecular representation, learned through MS/MS prediction tasks, can be further applied to enhance performance in other molecular-related tasks, such as predicting retention times (RT)"