Top n accuracy and auc metric calculation

top-n-accuracy-and-auc-metric-calculation

Summary

Compute ranking-based accuracy metrics (top-1, top-5, top-10, etc.) and area-under-curve (AUC) to evaluate how often a correct BGC-spectrum match ranks within the top n candidates, and assess discriminative power against a randomized baseline. This skill is essential for ranking-based retrieval tasks where a single ground-truth target must be re-identified within a sorted candidate list.

When to use

You have a ranked candidate list (e.g., BGCs sorted by IOKR or strain-correlation score) for each test spectrum, a known ground-truth BGC for each spectrum, and you want to measure retrieval performance across multiple recall depths (top-1 through top-200) and overall discrimination ability (AUC). This is appropriate when you cannot threshold on a single decision boundary and instead need to understand how ranking quality scales with acceptance of more candidates.

When NOT to use

• Input is a probabilistic classifier with soft confidence scores rather than deterministic ranks—use calibration and threshold-tuning instead.
• The ranking pool is smaller than ~50 candidates or contains many ties at the same score—top-n accuracy becomes unstable and AUC may not be interpretable.
• Ground truth is ambiguous (multiple equally valid BGCs per spectrum)—AUC computation requires binary labels and will produce spurious results.

Inputs

• ranked BGC candidate list per spectrum (spectrum ID, candidate BGC ID, score)
• ground-truth BGC-spectrum pairs (2966 validated links)
• candidate BGC pool (2242 BGCs with MIBiG homology structure assignments)

Outputs

• top-n accuracy table (n ∈ {1, 5, 10, 20, 200})
• AUC score and 95% confidence interval
• rank distribution histogram (with validated link positions overlay)
• p-value comparing validated vs. all-link score distributions

How to apply

For each of the 2966 BGC-spectrum pairs, rank all candidate BGCs (2242 with MIBiG structure assignments) by descending score. Record the rank position of the correct BGC. For each threshold n ∈ {1, 5, 10, 20, 200}, compute top-n accuracy as the proportion of spectra for which the correct BGC appears in positions 1 to n. For AUC, use the rank positions to construct a binary classification problem: for each spectrum, assign a positive label to the correct BGC and negative labels to all other ranked candidates, then compute area-under-the-receiver-operating-characteristic curve. Generate a randomized baseline by permuting BGC ranks for each spectrum independently and repeating the same calculations; valid IOKR performance must substantially exceed baseline AUC (0.5209 baseline vs. 0.6534 observed). Report mean ± variance of scores, p-values comparing validated vs. all-link distributions, and visual histograms overlaying validated-link positions on the full score distribution.

Related tools

• GNPS (source of MS2 spectra (6246 test spectra) matched to MIBiG BGCs via InChIKey for ground-truth pair construction)
• MIBiG (source of BGC candidate pool (2242 with structure assignments) and ground-truth validated BGC-spectrum pairs)
• NPLinker (framework orchestrating rank computation and metric calculation over BGC-spectrum pairs) — https://github.com/sdrogers/nplinker

Evaluation signals

• Top-n accuracy must be monotonically non-decreasing with increasing n (e.g., top-1 ≤ top-5 ≤ top-10); any decrease indicates a computational error.
• AUC must fall in [0.5, 1.0]; AUC > 0.5 indicates the scoring function ranks correct BGCs higher than random; AUC = 0.5 exactly suggests no discriminative power.
• Validated-link score distribution must have significantly higher mean than all-link distribution (p < 0.05), with visual separation in histograms.
• Randomized baseline AUC must be approximately 0.5 (±0.01); deviations suggest incorrect rank permutation or label assignment.
• Reported results must match published Table 3 and Figure S2 (IOKR top-1: 0.1208, top-5: 0.1708, AUC: 0.6534; baseline top-1: 0.0, top-5: 0.0014, AUC: 0.5209).

Limitations

• Performance breakdown by natural product compound class is difficult due to insufficient test set size for statistical stratification.
• IOKR applicability is restricted to BGCs with considerable homology to MIBiG entries; low-homology BGCs are not included in the candidate pool and cannot be ranked.
• Top-n accuracy is sensitive to candidate pool size and composition; results are not directly comparable across datasets with different numbers of structure-annotated BGCs.
• AUC computation assumes all negative candidates are equally undesirable; it does not account for semantic similarity among non-matching BGCs (e.g., same biosynthetic family).

Evidence

• [results] IOKR top-1 accuracy: 0.1208; top-5: 0.1708; top-10: 0.1870; top-20: 0.2121; top-200: 0.2946; AUC: 0.6534: "Table 3 shows the top-n performance of IOKR, i.e. how often the 'true' BGC match for a given spectrum is among the top n matches returned by IOKR: top-1: 0.1208, top-5: 0.1708, top-10: 0.1870"
• [results] random baseline top-1 accuracy: 0.0; top-5: 0.0014; top-10: 0.0044; top-20: 0.0103; top-200: 0.1486; AUC: 0.5209: "baseline score was estimated by randomising the rank of the structures for each spectrum, and the same process was repeated: top-1: 0.0, top-5: 0.0014, top-10: 0.0044, top-20: 0.0103, top-200:"
• [other] Calculate top-n accuracy (n=1,5,10,20,200) and area-under-curve (AUC) by comparing the rank of the correct BGC relative to all ranked candidates for each spectrum: "Calculate top-n accuracy (n=1,5,10,20,200) and area-under-curve (AUC) by comparing the rank of the correct BGC relative to all ranked candidates for each spectrum, and compare against a randomized"
• [other] IOKR mean score of 0.0105 for all 2966 BGC-spectrum links and 0.0364 for validated links (p=1.7968 × 10−9): "IOKR achieves a mean score of 0.0105 for all 2966 BGC-spectrum links and 0.0364 for validated links (p=1.7968 × 10−9)"
• [other] Construct 2966 BGC-spectrum pairs by matching MIBiG entries to GNPS spectra using the first part of the InChIKey: "Construct 2966 BGC-spectrum pairs by matching MIBiG entries to GNPS spectra using the first part of the InChIKey to avoid confounding by stereoisomerism."