Receiver operator characteristic curve generation

receiver-operator-characteristic-curve-generation

Summary

Construct ROC curves by ranking query spectra against a reference library and classifying hits as true or false positives based on structural identity (InChIKey), then computing and plotting true-positive rate versus false-positive rate across similarity score thresholds. This enables quantitative comparison of spectral similarity methods' discriminative power.

When to use

When you have computed similarity scores (cosine, modified cosine, Spec2Vec, or other metrics) between a set of query spectra and a reference library with known structural annotations (InChIKey), and you need to evaluate how well each scoring method distinguishes true structural matches from non-matches across a range of decision thresholds.

When NOT to use

• Input library spectra lack validated structural annotations (InChIKey). ROC curves require ground truth labels; without them, TP/FP classification is impossible.
• Similarity scores are not yet computed or pre-filtered. ROC generation requires a complete score matrix; pre-selection steps (e.g., precursor m/z matching) must occur first.
• Your goal is ranking or visualizing spectra in a network without classifier evaluation. If you only need to group similar spectra or draw molecular networks, direct similarity thresholding is sufficient; ROC curves add unnecessary overhead.

Inputs

• Query spectra (MS/MS data with precursor m/z, fragment peaks, and InChIKey annotation)
• Reference library spectra (MS/MS data with InChIKey annotation)
• Similarity score matrix (one row per query, one column per library spectrum; computed using cosine, modified cosine, Spec2Vec, or equivalent metric)

Outputs

• Receiver-operator-characteristic curve (plot with FPR on x-axis, TPR on y-axis)
• Area-under-curve (AUC) scalar per method
• True-positive rate and false-positive rate values at each threshold
• Ranked hit list with TP/FP classifications for each query–library pair

How to apply

For each similarity scoring method, rank query spectra by their computed similarity to each library spectrum. Classify each ranked hit as a true positive if the matched spectrum has an InChIKey within the first 14 characters (planar structure) that matches the query, or a false positive if the structure differs. Vary the classification threshold systematically across the observed similarity score range. At each threshold, calculate the true-positive rate (TP / (TP + FN)) and false-positive rate (FP / (FP + TN)). Plot FPR on the x-axis and TPR on the y-axis, then compute the area under the resulting curve (AUC) as a scalar summary metric. This allows direct comparison of method sensitivity and specificity trade-offs.

Related tools

• matchms (Computes cosine and modified cosine similarity scores; filters and preprocesses spectra before scoring) — https://github.com/matchms/matchms
• Spec2Vec (Computes Word2Vec-based spectral similarity embeddings for ROC comparison against cosine methods) — https://github.com/iomega/spec2vec
• Word2Vec (gensim) (Underlying embedding model trained on spectrum fragmentation patterns; used by Spec2Vec to generate similarity scores)
• scikit-learn or scipy.metrics (Computes ROC curve, AUC, and related metrics from binary classification labels and scores)

Examples

from sklearn.metrics import roc_curve, auc; fpr, tpr, thresholds = roc_curve(true_positive_labels, similarity_scores); roc_auc = auc(fpr, tpr); import matplotlib.pyplot as plt; plt.plot(fpr, tpr, label=f'ROC (AUC={roc_auc:.3f})'); plt.xlabel('False Positive Rate'); plt.ylabel('True Positive Rate'); plt.legend(); plt.show()

Evaluation signals

• ROC curve is monotonically increasing (TPR ≥ previous TPR for each FPR increase), and AUC is between 0 and 1.
• The number of true positives + false positives + true negatives + false negatives equals the total number of query–library pairs evaluated.
• AUC values for Spec2Vec are higher (>0.80) than for cosine and modified cosine methods, consistent with the paper's finding of better true/false-positive ratio.
• The curve passes through (0, 0) at the highest threshold (no hits classified as positive) and (1, 1) at the lowest threshold (all hits classified as positive).
• InChIKey matching is consistent: the same query–library pair always receives the same TP/FP label across repeated curve construction runs.

Limitations

• ROC curves require complete InChIKey annotations in both query and library spectra. Missing or incomplete annotations will bias TP/FP counts and lower apparent method performance.
• The choice of 'planar structure' (first 14 InChIKey characters) as ground truth is a proxy for structural similarity; some isomers may be incorrectly classified as non-matches if they differ only in stereochemistry.
• ROC construction assumes independence between query–library comparisons; in reality, spectra from the same compound will be more similar, violating this assumption and potentially inflating AUC.
• Spectral quality, pre-processing parameters (peak filtering, m/z tolerance, minimum matching peaks), and library composition directly affect ROC curves; comparisons across studies with different parameters may not be valid.

Evidence

• [other] For each similarity method, rank query spectra against library and classify hits as true positives (matching InChIKey within first 14 characters / planar structure) or false positives (non-matching structure) to construct receiver-operator-characteristic curves.: "rank query spectra against library and classify hits as true positives (matching InChIKey within first 14 characters / planar structure) or false positives (non-matching structure) to construct"
• [other] Calculate and report true-positive rate, false-positive rate, and area-under-curve for each method and compare against reported paper results.: "Calculate and report true-positive rate, false-positive rate, and area-under-curve for each method and compare against reported paper results"
• [other] Spec2Vec resulted in a notably better true/false positive ratio at all thresholds compared to cosine and modified cosine scores during library matching, achieving up to 88% accuracy and higher retrieval rates.: "Spec2Vec resulted in a notably better true/false positive ratio at all thresholds compared to cosine and modified cosine scores"
• [results] high Spec2Vec spectrum similarity score correlates stronger with structural similarity than the cosine or modified cosine scores: "high Spec2Vec spectrum similarity score correlates stronger with structural similarity than the cosine or modified cosine scores"
• [results] The poorer correlation between cosine and modified cosine similarity and structural similarity can largely be explained by high false positive rates: "The poorer correlation between cosine and modified cosine similarity and structural similarity can largely be explained by high false positive rates"