Benchmarking and comparative performance evaluation

Benchmarking and Comparative Performance Evaluation

Summary

Systematically compare a novel method (MIST-CF) against established baseline approaches (SIRIUS, FFN, Xformer) on standardized benchmark datasets using ranking accuracy metrics (top-1, top-3, top-k). This skill enables quantitative assessment of whether a new chemical formula inference model outperforms existing tools on retrospective and prospective MS/MS data.

When to use

Apply this skill when you have trained a new machine learning model for chemical formula or adduct assignment from MS/MS spectra and need to assess whether it offers genuine performance gains over established baselines. Use it specifically when you have access to ground-truth annotations (e.g., NPLIB1, NIST20, CASMI challenge datasets) and must justify publication or deployment claims with comparative accuracy metrics.

When NOT to use

• Input is a single spectrum or small set without ground-truth annotations — use case-by-case evaluation instead.
• Baseline tool (e.g., SIRIUS) is not available or cannot be run on your hardware/OS — benchmarking requires all competing methods executable in the same environment.
• You are comparing only internal model variants (e.g., ablation study) without a published external baseline — use internal ablation evaluation rather than full benchmarking.

Inputs

• MS/MS spectra in MGF format with precursor m/z values
• Ground-truth chemical formula and adduct type annotations
• Candidate formula lists (generated by SIRIUS decomp or chemical enumeration)
• Trained neural network weights for MIST-CF, FFN, Xformer baselines
• Preprocessed spectral data (normalized intensities, noise-filtered peaks)

Outputs

• Per-spectrum ranking predictions (ranked list of candidate formulas with adduct types)
• Top-1, top-3, top-k accuracy metrics for formula assignment
• Top-1, top-3, top-k accuracy metrics for adduct assignment
• Performance summary table (model × accuracy metric)
• Ranking accuracy plots (e.g., cumulative accuracy vs. rank position)
• Comparative analysis report (MIST-CF vs. SIRIUS, FFN, Xformer)

How to apply

Run inference predictions from all competing models (MIST-CF, SIRIUS decomp, FFN, Xformer) on the same test dataset, ensuring identical MS/MS preprocessing (intensity normalization, noise filtering, precursor m/z extraction). For each spectrum, compute ranked predictions for chemical formula and adduct type; evaluate using ranking metrics (top-1 accuracy: first ranked candidate matches ground truth; top-3, top-k: ground truth appears in top k predictions). Tabulate accuracy results side-by-side and generate ranking accuracy plots showing cumulative performance across the test set. On proprietary datasets (e.g., NIST20), train on disjoint splits; on public benchmarks (NPLIB1, CASMI22), use published train-test boundaries. Document hyperparameter tuning (via hyperopt or grid search) for all models on validation data to ensure fair comparison.

Related tools

• MIST-CF (Novel energy-based neural network model for ranking chemical formulas and adduct types from MS/MS spectra; main method under evaluation.) — https://github.com/samgoldman97/mist-cf
• SIRIUS (Baseline tool for formula enumeration via dynamic programming algorithm (SIRIUS decomp); generates candidate formula lists and provides ranking predictions for comparison.) — https://bio.informatik.uni-jena.de/software/sirius/
• FFN (Feed-forward neural network baseline model for formula ranking; included in comparative benchmarking experiments.)
• Xformer (Transformer baseline model for formula ranking; included in comparative benchmarking experiments.)
• SCARF (Related work on sinusoidal formula embeddings; used as feature representation in MIST-CF architecture.) — https://arxiv.org/abs/2303.06470

Examples

. run_scripts/benchmarking/train_mist_cf.sh && . run_scripts/benchmarking/train_ffn.sh && . run_scripts/benchmarking/train_xformer.sh && python run_scripts/benchmarking/eval_models.py

Evaluation signals

• Top-k accuracy values are monotonically non-decreasing as k increases (top-1 ≤ top-3 ≤ top-5, etc.).
• MIST-CF top-k accuracy meets or exceeds SIRIUS baseline on retrospective (NPLIB1, NIST20) and prospective (CASMI22) test sets, demonstrating improvement over fragmentation-tree-based method.
• Performance metrics are computed on a consistent test split with no data leakage between training and evaluation sets; hyperparameter tuning used only validation data.
• Adduct assignment accuracy is reported separately and is non-trivial (>baseline random selection of adduct types).
• Comparison tables show that all models were tuned to convergence (no model has obviously suboptimal hyperparameters relative to published results or validation curves).

Limitations

• MIST-CF trained on public NPLIB1 dataset may be less performant than NIST20-trained model (particularly for Orbitrap or higher-resolution MS/MS data), limiting generalizability claims.
• Benchmarking focuses on positive-mode MS/MS only; adduct types are still limited beyond [M+H]+, so evaluation does not cover negative-mode or diverse adduct scenarios.
• SIRIUS baseline still required for initial formula enumeration (candidate generation); MIST-CF learns ranking but does not replace formula discovery, limiting end-to-end de novo capability.
• Prospective evaluation (CASMI22) is a single challenge dataset; retrospective benchmarking on NPLIB1 may overfit to that data distribution.

Evidence

• [other] How does MIST-CF's end-to-end energy-based formula transformer perform on ranking chemical formula and adduct assignments compared to SIRIUS-based approaches on benchmark MS/MS data?: "research_question: How does MIST-CF's end-to-end energy-based formula transformer perform on ranking chemical formula and adduct assignments compared to SIRIUS-based approaches on benchmark MS/MS"
• [other] Evaluate ranking performance by computing top-1, top-3, and top-k accuracy metrics for formula and adduct predictions.: "4. Evaluate ranking performance by computing top-1, top-3, and top-k accuracy metrics for formula and adduct predictions."
• [other] Compare MIST-CF top-k accuracy results against SIRIUS baseline predictions on the same dataset.: "5. Compare MIST-CF top-k accuracy results against SIRIUS baseline predictions on the same dataset."
• [intro] MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases: "MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases"
• [intro] Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion: "Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion"
• [readme] Retrospective benchmarking: Benchmark MIST-CF performance with baseline models.: "Retrospective benchmarking Benchmark MIST-CF performance with baseline models."
• [readme] Prospective analysis on CASMI-2022: Compare MIST-CF and SIRIUS on CASMI-2022.: "Prospective analysis: CASMI 2022 Compare MIST-CF and SIRIUS on CASMI-2022."
• [readme] This model may be less performant than the model trained on the commercial NIST20 Library (particularly for Orbitrap or higher resolution data).: "This model may be less performant than the model trained on the commercial NIST20 Library (particularly for Orbitrap or higher resolution data)."
• [intro] Considering multiple adduct types beyond [M+H]+ (still only positive mode): "Considering multiple adduct types beyond [M+H]+ (still only positive mode)"