Regression performance metric computation

regression-performance-metric-computation

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

Compute standard regression performance metrics (mean absolute error, R², and other statistics) on model predictions against ground-truth labels to quantify generalizability of graph neural network collision cross section predictions. This skill validates whether a trained model reproduces reported performance benchmarks.

When to use

After generating collision cross section predictions on a validation or test set using a trained graph neural network model, and you need to quantify prediction accuracy and compare against reported performance metrics in the literature or prior experimental runs.

When NOT to use

• Predictions have not yet been generated on the test set—first run the model inference step.
• Test set labels are missing or misaligned with predictions (shape or index mismatch).
• You are evaluating classification performance rather than regression—use classification metrics (accuracy, F1, ROC-AUC) instead.

Inputs

• predicted_ccs_values (numeric array or column from predictions file)
• ground_truth_ccs_values (numeric column from test/validation dataset parquet file)
• test_set_metadata (dataset identifiers, e.g., 'ccsbase_3d.parquet', 'metlin_train_3d.parquet')

Outputs

• metrics_file (JSON or CSV with MAE, R², and other regression statistics)
• performance_summary (structured record of metric names and values for comparison against reported benchmarks)

How to apply

Load the model's predicted collision cross section values and the corresponding ground-truth CCS values from the test or validation dataset (typically stored in parquet format with columns like 'ccs' for ground truth and prediction output). Compute mean absolute error (MAE), R² (coefficient of determination), and any other regression statistics reported in the paper. Save the computed metrics to a structured output file (e.g., JSON or CSV) that records the metric names, values, and optionally dataset split metadata (train set, test set). The rationale is that regression metrics on held-out data directly measure model generalizability across different CCS databases (e.g., training on METLIN and testing on CCSBase reveals cross-database performance).

Related tools

• train-test.py (Generates predictions and can optionally compute metrics; orchestrates model evaluation workflow) — https://github.com/enveda/ccs-prediction
• sklearn.metrics (Python library providing mean_absolute_error, r2_score, and other regression metric functions)
• reproduce_figures notebooks (Jupyter notebooks that compute and visualize performance metrics for manuscript figures) — https://github.com/enveda/ccs-prediction/tree/main/notebooks/reproduce_figures

Examples

poetry run python scripts/train-test.py --prefix "train-metlin-test-ccsbase" --train-input-file "ccs-prediction/metlin_train_3d.parquet" --test-input-file "ccs-prediction/ccsbase_3d.parquet" --parameter-path "parameter/parameter-train-metlin-test-metlin.json" --model-output-file "model/train-metlin-test-metlin.h5" --ccs-column-name "ccs" --dropout-rate 0.1 --epochs 400

Evaluation signals

• Computed metrics match or closely reproduce the values reported in Engler et al. (2024) for each train/test dataset pair (e.g., 'train-metlin-test-ccsbase').
• Metrics file contains all expected fields: metric name (e.g., 'MAE', 'R2'), numeric value, and dataset split identifiers.
• R² is in range [−∞, 1] and MAE is non-negative; for good models on CCS prediction, R² should be > 0.8 and MAE should be < ~5 Ų (depending on database and adduct).
• Predictions and ground-truth arrays have matching lengths and no NaN/inf values in metric computation.
• Cross-database generalization comparison: metrics for 'train-metlin-test-ccsbase' should be interpretable as a measure of domain shift (expected to be worse than same-database splits).

Limitations

• Metrics are sensitive to data preprocessing and normalization; ensure training and test sets are processed identically (coordinates present/absent, SMILES canonicalization, adduct encoding).
• R² can be misleading if the test set distribution differs significantly from training (cross-database evaluation may show lower R² even if model is sound).
• No single metric fully captures generalizability; MAE and R² should be reported together with residual analysis and per-adduct or per-molecular-weight-bin breakdowns (see reproduce_figures notebooks).
• Pre-trained model weights and hyperparameters (dropout=0.1, epochs=400) must match those used to generate reported metrics; using different hyperparameters will produce non-comparable results.

Evidence

• [other] Compute reported performance metrics (e.g., mean absolute error, R², or other regression statistics) and save results to a metrics file.: "Compute reported performance metrics (e.g., mean absolute error, R², or other regression statistics) and save results to a metrics file."
• [readme] Predictions are available and can be directly downloaded from [...] The files should be unzipped and placed in the data directory.: "Predictions are available and can be directly downloaded from . The files should be unzipped"
• [readme] Run the notebooks located in the notebooks corresponding to each analysis. [...] reproduce_figures: the name of the notebooks indicates which notebook can reproduce which figures of the manuscript: "Run the notebooks located in the notebooks corresponding to each analysis. [...] reproduce_figures: the name of the notebooks indicates which notebook can reproduce which figures"
• [readme] prefix is used to generate the output files of the predictions of the test set: "prefix is used to generate the output files of the predictions of the test set"