High quality spectral library benchmarking

high-quality-spectral-library-benchmarking

Summary

Evaluate MS/MS spectral embedding model performance on curated, high-quality spectral libraries (MassBank and MassSpecGym) using hit@k retrieval metrics computed across multiple random query/reference splits. This skill assesses model robustness and generalization on spectral data with minimal quality artifacts.

When to use

You have a pre-trained MS/MS spectral embedding model and need to validate that it achieves strong and consistent retrieval performance on curated spectral libraries that represent real-world data quality standards. This is particularly important when the model was trained on larger, noisier datasets (e.g., GNPS, MoNA) and you want to verify generalization to high-quality references before deployment or publication.

When NOT to use

• If spectral data contains malformed or invalid SMILES strings—these must be removed beforehand via data cleaning steps
• If you are evaluating on training data or using the original training/test splits from MSBERT—only apply random splitting to test sets to avoid data leakage
• If your model was not trained with supervised contrastive learning or sinusoidal positional encoding—the reported baseline performance may not be representative

Inputs

• Pre-trained SpecEmbedding model checkpoint (Tanimoto supervised contrastive learning variant)
• Query spectra in MSP format
• Reference spectra in MSP format (from MassBank or MassSpecGym)
• 10 random query/reference split definitions (from figshare)
• Tokenizer configuration (vocabulary size 100, augmentation enabled)

Outputs

• hit@k metrics (k=1, 5, 10) per split
• Mean and standard deviation of hit@k across 10 splits
• Cosine similarity score matrix (query × reference)
• Top-k candidate indices per query spectrum

How to apply

Load the pre-trained SpecEmbedding model checkpoint (using Tanimoto supervised contrastive learning with sinusoidal positional encoding) and initialize a Tokenizer with vocabulary size 100 and augmentation enabled. Load query and reference spectra in MSP format from MassBank or MassSpecGym using read_raw_spectra(). Generate spectral embeddings using the embedding() function with batch size 512 and ModelTester inference wrapper. Compute pairwise cosine similarity scores between query and reference embeddings, then retrieve top-k candidate indices (k=1, 5, 10) using top_k_indices(). Calculate hit@k metrics by counting matches between query and reference compound identifiers. Repeat this evaluation across 10 random query/reference splits (available on figshare) and report both average hit@k scores and standard deviations. Compare results against reported performance ranges to confirm consistent generalization on high-quality spectra.

Related tools

• SpecEmbedding (Pre-trained deep learning model for MS/MS spectral embedding with sinusoidal positional encoding and supervised contrastive learning checkpoint) — https://github.com/sword-nan/SpecEmbedding
• ModelTester (Inference wrapper for generating spectral embeddings in batch mode with optional progress bar) — https://github.com/sword-nan/SpecEmbedding
• Tokenizer (Converts raw spectral peaks into token sequences (vocabulary size 100) with augmentation for model input) — https://github.com/sword-nan/SpecEmbedding
• SpecEmbedding-Comparison (Evaluation framework and benchmarking scripts for computing hit@k metrics and comparison with baseline methods) — https://github.com/sword-nan/SpecEmbedding-Comparison

Examples

from SpecEmbedding.utils.model import embedding, cosine_similarity, load_tanimoto_supcon_aug_model, top_k_indices
from SpecEmbedding.utils.clean import read_raw_spectra
from SpecEmbedding.trainer.trainer import ModelTester
from SpecEmbedding.data.tokenizer import Tokenizer
q_spectra = read_raw_spectra('./q.msp')
r_spectra = read_raw_spectra('./r.msp')
tokenizer = Tokenizer(100, True)
model = load_tanimoto_supcon_aug_model('cpu')
tester = ModelTester(model, 'cpu', False)
q_emb, _ = embedding(tester, tokenizer, 512, q_spectra, False)
r_emb, _ = embedding(tester, tokenizer, 512, r_spectra, False)
cosine_scores = cosine_similarity(q_emb, r_emb)
indices = top_k_indices(cosine_scores, 1)

Evaluation signals

• hit@k scores (especially hit@1, hit@5, hit@10) fall within or exceed the reported performance ranges for MassBank and MassSpecGym
• Standard deviations across 10 random splits are consistent with published ranges, indicating stable and reproducible retrieval performance
• Cosine similarity scores between matching query and reference spectra are substantially higher than non-matching pairs
• Top-k candidate indices correctly identify compound identifiers that match the query spectrum's reference compound
• Performance on curated libraries (MassBank, MassSpecGym) is comparable to or better than performance on noisier training datasets (GNPS, MoNA), confirming generalization

Limitations

• Evaluation relies on compound identifier matching; spectral variations of the same compound may not be captured if SMILES or compound ID fields are missing or inconsistent
• Performance is sensitive to data quality—spectra with malformed or invalid SMILES strings must be removed beforehand; results do not generalize to uncleaned spectral data
• Windows users may encounter numerical errors during cosine similarity computation due to @njit decorators from numba; requires commenting out decorators to run
• Hit@k metrics assume a single correct reference compound per query; isomers or structural analogs will be counted as false negatives
• Results are specific to the Tanimoto supervised contrastive learning checkpoint; other model variants or training procedures may yield different performance

Evidence

• [readme] We additionally tested MassBank and MassSpecGym, two curated spectral libraries. SpecEmbedding achieved consistently strong performance on these datasets as well.: "We additionally tested MassBank and MassSpecGym, two curated spectral libraries. SpecEmbedding achieved consistently strong performance on these datasets as well."
• [methods] Calculate hit@k metrics by counting matches between query and reference compound identifiers across 10 random query/reference splits from figshare.: "Calculate hit@k metrics by counting matches between query and reference compound identifiers across 10 random query/reference splits"
• [methods] Retrieve top-k candidate indices (k=1, 5, 10) from similarity scores using top_k_indices().: "Retrieve top-k candidate indices (k=1, 5, 10) from similarity scores using top_k_indices()."
• [methods] Verification: Verify hit@k scores and standard deviations match or fall within reported performance ranges demonstrating consistent generalization on high-quality curated spectra.: "Verify hit@k scores and standard deviations match or fall within reported performance ranges demonstrating consistent generalization on high-quality curated spectra"
• [readme] All cleaned data, along with the preprocessing scripts and 10-fold query/reference splits used for evaluation, are available on figshare: "All cleaned data, along with the preprocessing scripts and 10-fold query/reference splits used for evaluation, are available on figshare"
• [methods] Generate spectral embeddings for query and reference sets using the embedding() function with batch size 512 and ModelTester inference wrapper.: "Generate spectral embeddings for query and reference sets using the embedding() function with batch size 512 and ModelTester inference wrapper"
• [methods] Initialize Tokenizer with vocabulary size 100 and augmentation enabled.: "Initialize Tokenizer with vocabulary size 100 and augmentation enabled."
• [methods] Load pre-trained SpecEmbedding model using the Tanimoto supervised contrastive learning checkpoint with sinusoidal positional encoding via load_tanimoto_supcon_aug_model().: "Load pre-trained SpecEmbedding model using the Tanimoto supervised contrastive learning checkpoint with sinusoidal positional encoding"