Probability prediction metric computation

Probability-Prediction Metric Computation

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

Compute evaluation metrics (is_max and average_distance) that quantify how well predicted modification site probability distributions match ground-truth structural modifications. This skill bridges probabilistic predictions from spectral alignment to interpretable performance scores.

When to use

After running ModiFinder's probability generation on a known compound–modified compound pair, you have a vector of per-atom modification probabilities and need to validate whether the predicted probability peaks align with the true modification sites. Use this skill when you have both (1) predicted probabilities from mf.generate_probabilities() and (2) annotated structures with known modification locations.

When NOT to use

• Probabilities have not yet been generated (must run mf.generate_probabilities() first).
• Ground-truth modification sites are unknown or unlabeled in the input structures.
• Input probability vector length does not match the number of atoms in the target structure (will raise a dimension mismatch error).

Inputs

• Analog compound structure (RDKit Mol object or SMILES string)
• Modified compound structure (RDKit Mol object or SMILES string)
• Modification site probability vector (1D array of floats, length = number of atoms)

Outputs

• is_max score (float, 0–1; 1.0 if highest probability atom is a true modification site)
• average_distance score (float, typically 0–1; 1.0 if predictions perfectly match ground truth)
• Evaluation results JSON (includes both scores and intermediate probability data)

How to apply

Instantiate BasicEvaluationEngine and call evaluate_single() with three arguments: the analog (known) structure, the target (modified) structure, and the probability vector. The engine computes two complementary metrics: is_max checks whether the highest probability is assigned to a true modification site (binary/presence metric), and average_distance measures the mean graph-edit distance from predicted high-probability atoms to the nearest true modification atom (continuous/proximity metric). Run the evaluation twice—once with default_method='average_distance' and once with evaluation_method='is_max'—to obtain both scores. The scores range from 0–1, with 1.0 indicating perfect alignment of predictions to ground truth; average_distance penalizes off-target predictions by their topological distance in the molecular graph.

Related tools

• ModiFinder (Generates modification site probabilities via generate_probabilities(); required upstream step before metric computation) — https://github.com/Wang-Bioinformatics-Lab/ModiFinder_base
• BasicEvaluationEngine (Core evaluation component that computes is_max and average_distance scores from structures and probabilities) — https://github.com/Wang-Bioinformatics-Lab/ModiFinder_base
• RDKit (Molecular structure representation and graph-based distance calculations used by the evaluation engine) — http://www.rdkit.org/
• Python (Runtime environment for instantiating and invoking BasicEvaluationEngine)

Examples

from modifinder import BasicEvaluationEngine; evaluator = BasicEvaluationEngine(default_method='average_distance'); is_max_score = evaluator.evaluate_single(analog_structure, target_structure, probabilities, evaluation_method='is_max'); avg_dist_score = evaluator.evaluate_single(analog_structure, target_structure, probabilities, evaluation_method='average_distance')

Evaluation signals

• Both is_max and average_distance scores are in the valid range [0.0, 1.0].
• is_max = 1.0 when the atom with maximum probability is a true modification site; is_max < 1.0 otherwise.
• average_distance scores inversely correlate with the graph-edit distance between predicted and true modification atoms; closer topological proximity yields higher scores.
• Probability vector sums to a normalized value consistent with the alignment engine's output (check intermediate JSON for consistency).
• Results are reproducible: identical structures and probability inputs yield identical metric outputs across multiple invocations.

Limitations

• Requires ground-truth annotation of modification sites in both input structures; metric cannot be computed if true sites are ambiguous or unknown.
• average_distance metric depends on molecular graph topology and atom indexing; structural isomers or tautomers with identical molecular formula may yield different scores despite similar chemistry.
• Performance is sensitive to the quality and specificity of the upstream probability generation (ModiFinder's alignment and annotation); garbage probabilities will produce low scores regardless of evaluation engine correctness.
• The metric assumes a single primary modification site; compounds with multiple simultaneous modifications may have diluted probability mass, reducing both scores even if predictions are chemically plausible.

Evidence

• [other] research_question excerpt from task_id=task_002: "What are the BasicEvaluationEngine scores (is_max and average_distance) for ModiFinder's modification site predictions under the default Modified Cosine + MAGMa condition?"
• [other] workflow step 3 and 4 from task_id=task_002: "Generate modification site probabilities via mf.generate_probabilities(). 4. Instantiate BasicEvaluationEngine with default_method='average_distance'."
• [other] workflow step 5 from task_id=task_002: "Call evaluate_single(analog_structure, target_structure, probabilities) to compute average_distance score; call again with evaluation_method='is_max' to compute is_max score."
• [other] finding from task_id=task_002: "ModiFinder achieves BasicEvaluationEngine scores of is_max = 1.0 and average_distance = 0.514 under the default Modified Cosine + MAGMa condition."
• [readme] Core API documentation from README: "Pass these to a ModiFinder Object ... Collect dictionary results: peaksObj, fragmentsObj = siteLocator.get_result()"