Machine learning hyperparameter extraction

Machine-Learning Hyperparameter Extraction

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

Extract and validate optimized hyperparameter values (e.g., nIter, method) for trained machine-learning classifiers from cross-validation output. This skill identifies which hyperparameter combinations were selected during model tuning and documents them for reproducibility and model retraining.

When to use

You have completed cross-validation tuning of one or more machine-learning models (e.g., AdaBoost, SVM, Random Forest) on a development dataset using the caret package and need to identify which specific hyperparameter values were selected as optimal. This is necessary before training a final classifier on the full development set or before applying the model to held-out test data.

When NOT to use

• You are using pre-trained models from external sources without access to their cross-validation history or prediction data frames.
• Hyperparameter tuning was not performed; the model uses default hyperparameters and no runCrossValidation() output is available.
• The evaluation measures output is unavailable or does not align with the pred data frame (mismatched cross-validation runs).

Inputs

• List object returned by runCrossValidation() containing trained models
• Prediction data frame from models$[AlgorithmName]$pred
• Evaluation measures data frame from getEvaluationMeasures()
• Cross-validation parameters (k-fold number, repetition number, metric set)

Outputs

• Extracted hyperparameter combinations (e.g., nIter, method values)
• Identified optimized hyperparameter set for best-performing model configuration
• Validated correspondence between hyperparameters and evaluation metrics

How to apply

After running runCrossValidation() with specified k-fold and repetition parameters (e.g., k=5, repNum=10) and a selected metric set (e.g., metricSet=c('M4','M7','M11')), access the prediction data frame returned by the trained model object (e.g., models$AdaBoost_M11$pred). Extract the unique hyperparameter columns (nIter, method, etc.) from this pred data frame to identify all candidate hyperparameter combinations tested. Cross-reference the extracted combinations against the evaluation measures output (generated by getEvaluationMeasures()) to confirm which hyperparameter set corresponds to the best-performing configuration according to your chosen evaluation metric. Document the optimized values for use in the subsequent trainClassifier() call on the full development set.

Related tools

• caret (R package used to implement machine-learning algorithms and manage cross-validation hyperparameter tuning)
• MetaClean (Peak-quality classifier package providing wrapper functions runCrossValidation(), getEvaluationMeasures(), and trainClassifier() that manage hyperparameter tuning and extraction workflow) — https://github.com/KelseyChetnik/MetaClean
• R (Statistical computing environment in which hyperparameter extraction, data frame manipulation, and validation are performed)

Examples

models <- runCrossValidation(trainData = pqm_development, k = 5, repNum = 10, rand.seed = 512, models = 'all', metricSet = c('M4','M7','M11')); hyperparams <- unique(models$AdaBoost_M11$pred[, c('nIter', 'method')]); eval_measures <- getEvaluationMeasures(models); best_idx <- which.max(eval_measures$Accuracy); optimal_hyperparams <- hyperparams[best_idx, ]

Evaluation signals

• The extracted hyperparameter columns (nIter, method) are present and non-null in the pred data frame.
• The unique hyperparameter combinations extracted match the set of combinations that were iterated during cross-validation (no spurious or missing values).
• The optimized hyperparameter set identified can be cross-referenced to the row(s) in the evaluation measures output with the best (highest or lowest, depending on metric direction) performance score.
• The hyperparameter values are numeric or character strings matching the data types expected by trainClassifier() (e.g., nIter is an integer, method is a character string such as 'Adaboost.M1').
• When the optimized hyperparameters are passed to trainClassifier() on the full development set, model training succeeds without parameter-type or range errors.

Limitations

• The article does not provide the hyperparameter values in the reported text; extraction relies entirely on programmatic access to the pred data frame, which is not published in the paper.
• Cross-validation outputs are sensitive to random seed and sampling strategy; reproducibility requires recording the exact runCrossValidation() parameters (k, repNum, rand.seed) used during tuning.
• If multiple hyperparameter combinations achieve similar evaluation scores, the extraction method does not disambiguate; practitioners must apply domain knowledge or additional criteria to select among tied candidates.
• The skill applies to MetaClean's wrapper functions and caret-compatible algorithms; extraction workflows may differ for other machine-learning frameworks or packages.

Evidence

• [other] Load the trained models list returned by runCrossValidation() with parameters k=5, repNum=10, rand.seed=512, models='all', metricSet=c('M4','M7','M11') on pqm_development (500 peaks, 89 samples). 2. Access the prediction data frame from models$AdaBoost_M11$pred. 3. Extract the unique hyperparameter combinations (nIter and method columns) from the pred data frame.: "Access the prediction data frame from models$AdaBoost_M11$pred. 3. Extract the unique hyperparameter combinations (nIter and method columns) from the pred data frame."
• [other] Identify and record the optimized hyperparameters determined during cross-validation (nIter=150, method='Adaboost.M1' per the example). 5. Validate that these hyperparameters correspond to the best-performing AdaBoost configuration by cross-referencing with evaluation measures output.: "Validate that these hyperparameters correspond to the best-performing AdaBoost configuration by cross-referencing with evaluation measures output."
• [methods] The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms.: "The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms."
• [methods] We use the getEvaluationMeasures function to do this.: "We use the getEvaluationMeasures function to do this."
• [methods] metaclean_model <- trainClassifier(trainData = pqm_development, model = "AdaBoost", metricSet = "M11", hyperparameters = hyperparameters): "trainClassifier(trainData = pqm_development, model = "AdaBoost", metricSet = "M11", hyperparameters = hyperparameters)"