Mass spectrometry benchmark analysis

mass-spectrometry-benchmark-analysis

Summary

Quantitatively evaluate chemical formula ranking accuracy and adduct assignment performance across MS/MS inference methods using standardized benchmark datasets and ranking metrics. This skill isolates the contribution of individual model innovations (e.g., multi-adduct support) by computing top-k accuracy deltas against controlled baseline modes.

When to use

You have implemented or modified a tandem mass spectrometry formula inference model and need to measure whether a specific architectural change (e.g., enabling multi-adduct support, switching from fragmentation trees to transformer networks) improves formula-adduct ranking accuracy relative to a restricted baseline or competing tool. Apply this skill when you have a published or standardized benchmark dataset with ground-truth formula and adduct annotations.

When NOT to use

• Input spectra lack corresponding ground-truth formula or adduct labels — benchmarking requires annotated reference data.
• You are evaluating model performance on the same dataset used for training; use a held-out test or external benchmark (e.g., CASMI22) to avoid inflated accuracy estimates.
• Your goal is to rank spectra by confidence or filter low-quality spectra rather than compare competing formula ranking models — use filtering or scoring workflows instead.

Inputs

• MS/MS spectra in MGF or equivalent format
• Benchmark dataset with ground-truth formula annotations
• Ground-truth adduct type assignments
• Model weights or trained checkpoint
• Configuration YAML specifying inference mode (e.g., single vs. multi-adduct)

Outputs

• Ranked formula predictions (scored list per spectrum)
• Ranked adduct assignments per spectrum
• Top-1 and top-k accuracy metrics for each mode
• Performance delta report (accuracy improvement by feature)
• Per-spectrum and aggregate comparison table

How to apply

Load a benchmark dataset (e.g., NPLIB1, NIST20, or CASMI22) with MS/MS spectra annotated with correct formula and adduct type. Run inference on the same spectra twice: once in restricted mode (e.g., [M+H]+ only) and once in full mode (e.g., multi-adduct energy-based ranking). For each run, record ranked formula-adduct pair predictions. Compute ranking accuracy metrics (top-1 and top-k correct pair identification rates) for both modes. Calculate the performance delta (full mode accuracy minus restricted mode accuracy) to isolate the contribution of the specific feature. Generate a comparison report listing accuracy improvements, failure modes, and dataset-specific patterns.

Related tools

• MIST-CF (End-to-end energy-based chemical formula inference model supporting multi-adduct ranking; used to generate ranked predictions in full-mode and restricted-mode configurations for delta comparison.) — https://github.com/samgoldman97/mist-cf
• SIRIUS (Baseline formula enumeration and fragmentation tree method used for comparison and to generate candidate formula lists via SIRIUS decomp dynamic programming.) — https://bio.informatik.uni-jena.de/software/sirius/
• SCARF (Sinusoidal formula embedding technique integrated into MIST-CF architecture; enables comparison of embedding-based vs. tree-based formula ranking.) — https://arxiv.org/abs/2303.06470

Examples

python run_scripts/benchmarking/eval_models.py --config configs/mist_cf_canopus.yaml --test_data data/canopus_train/test_split.csv --output_dir results/ --restricted_mode [M+H]_only

Evaluation signals

• Top-1 accuracy (fraction of spectra with correct formula-adduct pair ranked first) is higher in full mode than restricted mode; delta is quantifiable and statistically meaningful.
• Top-k accuracy curves (k=1–5, 1–10) are monotonically increasing and show steeper improvement in full mode, indicating the feature enables better ranking discrimination.
• Per-spectrum rankings are reproducible: running inference twice on the same spectrum produces identical ranked lists and scores.
• Formula-adduct pairs in the top-k predictions are chemically plausible (correct monoisotopic mass, consistent with adduct ionization rules) even when not top-1.
• Performance delta is consistent across dataset partitions (e.g., multiple folds, different adduct subsets, different m/z ranges) and does not reflect overfitting to a single spectrum subset.

Limitations

• Multi-adduct support in MIST-CF is limited to positive-mode ionization; negative-mode or mixed-polarity datasets will not benefit from adduct diversity.
• Benchmark performance on public data (NPLIB1) may not transfer to proprietary datasets (NIST20) due to spectrum preprocessing, instrument type, and intensity distribution differences; model trained on NIST20 is reportedly more performant but available only under license.
• Ranking accuracy depends on the candidate formula list enumerated by SIRIUS decomp; if the ground-truth formula is not in the candidate set, both restricted and full modes will fail to rank it, masking upstream enumeration errors.
• Top-k metrics require a choice of k; small k values (k=1–3) may underestimate practical utility for high-throughput screening where top-10 predictions are acceptable.

Evidence

• [other] What is the performance improvement in chemical formula ranking accuracy when MIST-CF incorporates support for multiple adduct types compared to restricting predictions to [M+H]+ only?: "What is the performance improvement in chemical formula ranking accuracy when MIST-CF incorporates support for multiple adduct types compared to restricting predictions to [M+H]+ only?"
• [other] Run MIST-CF inference in [M+H]+-only mode, restricting the model to rank formulas without multi-adduct support, and record ranked formula predictions and adduct assignments.: "Run MIST-CF inference in [M+H]+-only mode, restricting the model to rank formulas without multi-adduct support, and record ranked formula predictions and adduct assignments."
• [other] Compute ranking accuracy metrics (top-1 and top-k correct formula-adduct pair identification) for both modes.: "Compute ranking accuracy metrics (top-1 and top-k correct formula-adduct pair identification) for both modes."
• [other] Calculate the performance delta (multi-adduct accuracy minus [M+H]+-only accuracy) to isolate MULTI_ADDUCT_SUPPORT contribution and generate a comparison report.: "Calculate the performance delta (multi-adduct accuracy minus [M+H]+-only accuracy) to isolate MULTI_ADDUCT_SUPPORT contribution and generate a comparison report."
• [readme] MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases: "MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases"
• [readme] Considering multiple adduct types beyond [M+H]+ (still only positive mode): "Considering multiple adduct types beyond [M+H]+ (still only positive mode)"
• [other] Load the published benchmark dataset containing MS/MS spectra with known formula and adduct ground truth.: "Load the published benchmark dataset containing MS/MS spectra with known formula and adduct ground truth."
• [other] Run MIST-CF inference in full multi-adduct mode using the complete energy-based modeling approach, allowing ranking across multiple positive-mode adduct types, and record ranked predictions.: "Run MIST-CF inference in full multi-adduct mode using the complete energy-based modeling approach, allowing ranking across multiple positive-mode adduct types, and record ranked predictions."