Retention order prediction model execution

retention-order-prediction-model-execution

Summary

Execute pre-configured retention-order prediction models (RankSVM or SVR) on liquid-chromatography metabolite datasets to reproduce published ranking predictions and performance metrics. This skill validates that experimental scripts and dependencies are properly installed and configured to generate predicted retention orders comparable to ground-truth orderings.

When to use

You have access to the aalto-ics-kepaco/retention_order_prediction repository, have installed all Python (scipy, numpy, sklearn, joblib, pandas, networkx) and R dependencies, possess molecular feature data (MACCS fingerprints or equivalent), and need to execute a specific evaluation scenario (baseline_single, all_on_one, etc.) to generate ranked metabolite orderings for a target chromatographic system.

When NOT to use

• Input molecular structures or retention data are not from liquid-chromatography experiments; this skill is specific to LC retention order prediction.
• Required Python or R dependencies are not installed or versions do not meet minimums (Python 3.5–3.6, scipy >= 0.19.1, sklearn >= 0.19.0); the skill will fail at import time.
• MACCS fingerprints or other configured molecular features have not been pre-computed; the script expects feature matrices as input, not raw chemical structures.

Inputs

• evaluation_scenarios_main.py script from repository
• JSON configuration file (e.g., results/raw/PredRet/v2/config.json) specifying molecular representation kernel, fingerprint predictor, and feature scaler
• Molecular feature matrices (MACCS counting fingerprints or binary MACCS) pre-computed for all metabolites in the dataset
• Ground-truth retention-order annotations for one or more chromatographic systems
• Installed Python environment with scipy >= 0.19.1, numpy >= 1.13.1, sklearn >= 0.19.0, joblib >= 0.11, pandas >= 0.20.3, networkx >= 2.0

Outputs

• Predicted retention orders (ranked lists of metabolites) for the target evaluation system
• Performance metrics: ranking accuracy, Spearman correlation coefficient, other rank-based scores
• Results directory hierarchy containing model serializations, hyperparameter logs, and comparison tables (predicted vs. ground-truth orderings)
• HTML summary report (ECCB2018.html) generated by R post-processing scripts showing aggregated results across all scenarios

How to apply

Invoke evaluation_scenarios_main.py with seven parameters: the estimator type (ranksvm or svr), the evaluation scenario (baseline_single, baseline_single_perc, all_on_one, all_on_one_perc, or met_ident_perf_GS_BS), the system set (10, imp, or 10_imp), target system index (−1 for all systems or 0 to |sysset|−1 for a specific system), path to a JSON configuration file specifying molecular features (kernel type, fingerprint predictor, feature scaler), the number of parallel jobs for hyperparameter search, and a debug flag. The script internally performs cross-system training, hyperparameter optimization, prediction on the evaluation system, and writes results to a hierarchical output directory. Compare predicted retention orders against ground-truth orderings using ranking metrics (ranking accuracy, Spearman correlation) stored in the output directory.

Related tools

• evaluation_scenarios_main.py (Primary entry point script that orchestrates estimator selection, scenario configuration, hyperparameter search, model training, and prediction execution) — github.com/aalto-ics-kepaco/retention_order_prediction
• RankSVM (Order predictor estimator for ranking retention order via pairwise preference learning) — github.com/aalto-ics-kepaco/retention_order_prediction
• SVR (Support Vector Regression) (Alternative order predictor estimator for continuous retention prediction) — github.com/aalto-ics-kepaco/retention_order_prediction
• model_selection_cls.py (Hyperparameter optimization and cross-validation module invoked during model training) — github.com/aalto-ics-kepaco/retention_order_prediction
• rcdk / rcdkTools (R package for MACCS and other counting fingerprint calculation via Chemical Development Kit (CDK)) — github.com/bachi55/rcdkTools
• OpenBabel (Command-line tool (obabel >= 2.3.2) for molecular structure preprocessing (optional, only if data preprocessing is repeated)) — http://openbabel.org/wiki/Main_Page

Examples

python src/evaluation_scenarios_main.py ranksvm baseline_single 10 -1 results/raw/PredRet/v2/config.json 2 False

Evaluation signals

• Script exits without import errors and dependency resolution warnings; verify presence of all required packages (scipy, numpy, sklearn, joblib, pandas, networkx, matplotlib optional) at runtime.
• Output directory hierarchy is created at expected location (results/PredRet/v2/final/[estimator]/[config_hash]/[feature_representation]) containing serialized model files, hyperparameter logs, and prediction results.
• Predicted retention orders can be loaded and parsed as ranked metabolite lists; each list contains the same metabolites as the ground-truth annotation with no missing or duplicate entries.
• Ranking accuracy and Spearman correlation metrics are computed and fall within reasonable ranges (0–1 for accuracy, −1 to +1 for Spearman ρ); metrics are consistent with published Table 3 or Table 4 results when using matching configuration and scenario parameters.
• HTML summary report (results/ECCB2018.html) is successfully generated by R post-processing; results can be opened in a web browser and tabular comparisons (predicted vs. ground-truth) are human-readable and match numeric outputs from Python phase.

Limitations

• Scripts are tested only on Python 3.5–3.6; behavior on newer Python versions (3.7+) is not documented and may encounter package compatibility issues.
• No changelog is provided in the repository, so incremental updates or bug fixes to the codebase cannot be tracked or verified.
• Data preprocessing (MACCS fingerprint calculation) requires additional R packages (data.table, ggplot2, knitr, rcdk, Matrix, obabel2R, fingerprint) and the OpenBabel command-line tool; this introduces external dependency brittleness beyond Python package management.
• Configuration is stored in JSON files; incorrect or missing keys in the configuration will cause silent failures or mismatched hyperparameter ranges rather than explicit validation errors.

Evidence

• [readme] All experiments of the paper can be reproduced by using the evaluation_scenarios_main.py script with the proper parameters: "All experiments of the paper can be reproduced by using the evaluation_scenarios_main.py script with the proper parameters"
• [readme] There is no further installation required. The scripts run out of the box, if all the package dependencies are satisfied.: "There is no further installation required. The scripts run out of the box, if all the package dependencies are sattisfied."
• [readme] The following packages are required: scipy >= 0.19.1, json >= 2.0.9, numpy >= 1.13.1, joblib >= 0.11, pandas >= 0.20.3, sklearn >= 0.19.0, networkx >= 2.0, matplotlib >= 2.1 (optional): "The following packages are required: scipy >= 0.19.1, json >= 2.0.9, numpy >= 1.13.1, joblib >= 0.11, pandas >= 0.20.3, sklearn >= 0.19.0, networkx >= 2.0, matplotlib >= 2.1 (optional)"
• [other] Compare predicted retention orders against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar).: "Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar)."
• [readme] The results will be stored into: results/PredRet/v2/final/ranksvm_slacktype=on_pairs/allow_overlap=True_d_lower=0_d_upper=16_ireverse=False_type=order_graph/difference/maccsCount_f2dcf0b3/minmax/baseline_single: "The results will be stored into: results/PredRet/v2 └── final └── ranksvm_slacktype=on_pairs └── allow_overlap=True_d_lower=0_d_upper=16_ireverse=False_type=order_graph └── difference └──"