Baseline comparison performance benchmarking

baseline-comparison-performance-benchmarking

Summary

Quantitatively compare a candidate annotation method against a reference baseline by computing precision, recall, and coverage metrics on the same spectral dataset, then calculating relative improvement as a percentage to verify achievement of a target performance threshold. This skill validates whether a novel approach (e.g., neural network fingerprint regeneration) delivers claimed gains over simpler methods (e.g., spectral library matching alone).

When to use

When you have implemented a novel annotation algorithm or network and must verify it outperforms a simpler reference approach on held-out or independent spectral data. Specifically: (1) you have ground-truth annotations for a common test set; (2) you can run both the baseline method and the candidate method on that same set; (3) you have a quantitative success criterion (e.g., '~20% recall boost') and need to measure whether it is met. Do NOT use this skill if you lack ground-truth labels or cannot execute both methods on identical inputs.

When NOT to use

• Input spectral dataset lacks ground-truth lipid annotations; baseline and candidate methods are applied to different subsets of data.
• Baseline method has not been benchmarked or validated independently; its reference performance is unknown or unstable.
• Candidate method is designed for a different ionization mode, lipid class, or MS instrument type than the baseline; direct comparison is confounded by these factors.

Inputs

• MS/MS spectral data in mzML format (or preprocessed .rda format after XCMS/RaMS preprocessing)
• Initial baseline annotations from spectral library matching (e.g., EQ module output)
• Ground-truth lipid identities with structural metadata (chain composition, double bond positions)
• Candidate method predictions with confidence scores or probability outputs

Outputs

• Baseline recall, precision, coverage scores
• Candidate method recall, precision, coverage scores
• Relative improvement percentages for each metric
• Boolean pass/fail outcome against target threshold
• Confidence intervals or cross-validation fold statistics

How to apply

Load processed MS/MS spectral data with initial (baseline) annotations from spectral library matching. Execute the baseline annotation workflow to obtain baseline recall, precision, and coverage. Then execute the candidate method (e.g., Wide-spectrum Modeling Yield network) on the same spectral inputs to obtain candidate predictions with confidence scores. For each metric, compute the relative improvement as (candidate_metric − baseline_metric) / baseline_metric × 100%. Compare the observed improvement against the target threshold (e.g., 20% recall boost). Document whether the threshold is met and report 95% confidence intervals or cross-validation folds to assess reproducibility. Ground-truth lipid identities must be consistent across both runs to ensure fair comparison.

Related tools

• XCMS (Processes mass spectrometry data for nonlinear peak alignment, matching, and initial peak detection prior to baseline annotation) — https://bioconductor.org/packages/XCMS/
• CAMERA (Integrated strategy for compound spectra extraction and annotation of LC/MS data to support spectral library matching baseline) — https://bioconductor.org/packages/CAMERA/
• LipidIN EQ Module (Performs spectral querying against lipid fragmentation library to generate baseline annotations for comparison) — https://github.com/LinShuhaiLAB/LipidIN
• LipidIN Wide-spectrum Modeling Yield (WMY) network (Candidate method that regenerates lipid fingerprints to produce enhanced predictions compared to baseline spectral matching) — https://github.com/LinShuhaiLAB/LipidIN

Examples

# Load baseline annotations and candidate WMY predictions, then compute metrics
recall_baseline <- sum(baseline_match & truth) / sum(truth)
recall_wmy <- sum(wmy_match & truth) / sum(truth)
improvement_pct <- (recall_wmy - recall_baseline) / recall_baseline * 100
if (improvement_pct >= 20) { print("Target 20% recall boost ACHIEVED") } else { print("Target NOT met") }

Evaluation signals

• Baseline and candidate methods produce scores for all samples in the test set without errors or missing values; no samples are excluded from either pipeline.
• Recall, precision, and coverage metrics are computed identically for both baseline and candidate (e.g., same denominator definition, same ground-truth label source).
• Relative improvement percentage for the primary metric (e.g., recall) meets or exceeds the stated target threshold (e.g., ≥20% boost); cross-validation or bootstrap confidence intervals do not overlap zero.
• Precision and coverage do not degrade substantially in the candidate method; if one metric improves, others are held or improve in parallel (no single-metric cherry-picking).
• Results are reproducible across multiple random seeds, cross-validation folds, or independent test cohorts; no overfitting to the evaluation set is evident.

Limitations

• Comparison is valid only for lipid classes and ionization modes (positive, negative [M+CH3COO]−, [M+COOH]−) represented in both baseline and candidate training data; performance gains may not generalize to unseen lipid types or instrument platforms.
• Ground-truth annotations must be curated and internally consistent; errors or ambiguities in ground truth propagate to both baseline and candidate measurements, potentially masking or inflating true performance differences.
• The Wide-spectrum Modeling Yield network requires pre-trained weights or sufficient labeled data for training; if weights are unavailable or training data is small, candidate method performance may degrade, yielding unfair comparison.
• Relative improvement is a ratio metric and can be misleading if baseline performance is already very high (e.g., 95% recall); report absolute improvements and effect sizes alongside percentages.
• Cross-species or multi-instrument validation is not provided in the LipidIN README; claimed 20% recall boost is observed in reported experiments but reproducibility on independent cohorts is not yet established.

Evidence

• [other] Does applying the Wide-spectrum Modeling Yield (WMY) network to regenerate lipid fingerprints produce the reported ~20% estimated recall boost in lipid annotation coverage and accuracy?: "research_question: Does applying the Wide-spectrum Modeling Yield (WMY) network to regenerate lipid fingerprints produce the reported ~20% estimated recall boost in lipid annotation coverage and"
• [other] Compare regenerated fingerprints against baseline spectral-matching-only annotations to calculate recall, precision, and coverage metrics. Compute estimated recall boost as (recall_wmy − recall_baseline) / recall_baseline × 100% and verify achievement of ~20% improvement threshold.: "4. Compare regenerated fingerprints against baseline spectral-matching-only annotations to calculate recall, precision, and coverage metrics. 5. Compute estimated recall boost as (recall_wmy −"
• [readme] Wide-spectrum Modeling Yield network for regenerating lipid fingerprints to further improve coverage and accuracy with a 20% estimated recall boosting: "LipidIN integrates a Wide-spectrum Modeling Yield network for regenerating lipid fingerprints to further improve coverage and accuracy with a 20% estimated recall boosting."
• [readme] The application of LipidIN in multiple tasks demonstrated reliability and potential for lipid annotation and biomarker discovery.: "The application of LipidIN in multiple tasks demonstrated reliability and potential for lipid annotation and biomarker discovery."