Cross validation model aggregation

cross-validation-model-aggregation

Summary

Aggregate learned weight matrices from multiple cross-validated neural network models to construct a robust, consensus feature attribution score matrix that captures microbe-metabolite interaction patterns across model replicates. This skill isolates the signal in learned relationships from training variation by pooling weights across all CV folds and iterations.

When to use

After training multiple neural network models via k-fold cross-validation (e.g., 10 iterations × 10-fold = 100 models), when you need to extract and aggregate learned feature interactions rather than relying on a single trained model. Essential when the goal is interpretability of feature relationships (e.g., which microbes drive metabolite prediction) rather than point prediction alone, and when you want to reduce overfitting artifacts by averaging signals across held-out folds.

When NOT to use

• Input is a single trained model, not an ensemble from cross-validation (no aggregation needed).
• Linear regression models (e.g., Elastic Net, MelonnPan) are used instead of neural networks; Olden's method is specific to neural network weight products.
• The goal is prediction accuracy only; feature attribution aggregation adds no value for point prediction and incurs computational overhead.

Inputs

• trained neural network models (100 total from 10 CV iterations × 10-fold CV)
• weight matrices W_l from all hidden layers of each trained model
• background distribution of attribution scores from shuffled cross-validation
• 95th percentile SCC threshold identifying well-predicted metabolites
• 97.5th percentile threshold for significant feature attribution scores

Outputs

• per-model feature attribution score matrices S_i (one per trained network, indexed by microbe and metabolite feature names)
• aggregated microbe-metabolite attribution score matrix (rows = microbes, columns = metabolites, values in [−1, 1])
• normalized and clipped feature attribution matrices (NumPy arrays or CSV tables)
• binary significance mask (elements with |value| ≥ 97.5th percentile threshold)

How to apply

Collect all trained neural network weight matrices W_l from every hidden layer l across all 100 cross-validated models (10 iterations of 10-fold CV). For each model, apply Olden's method by computing the product of weight matrices: S = ∏(W_l) for all layers l, where rows represent input features (microbes) and columns represent output features (metabolites). Record the resulting per-model attribution matrix S_i. Normalize each matrix by dividing element-wise by the significant threshold (97.5th percentile) derived from a background distribution generated by shuffling samples during CV. Clip all values to [−1, 1] to bound attribution scores. The final aggregated attribution matrix represents consensus microbe-metabolite interaction strengths, with positive scores indicating increased abundance of microbe→increased metabolite, and negative scores indicating inverse relationships.

Related tools

• MiMeNet (multi-layer perceptron neural network framework that trains 10 CV iterations of 10-fold models and applies this aggregation skill to construct microbe-metabolite feature attribution matrices) — https://github.com/YDaiLab/MiMeNet
• TensorFlow / PyTorch (neural network training and weight matrix extraction)
• NumPy (matrix multiplication (weight product computation), normalization, clipping, and storage of attribution matrices)
• Scikit-learn (cross-validation fold generation and model ensemble orchestration)

Examples

# Extract and aggregate weights from 100 cross-validated MiMeNet models
for model_idx in range(100):
    model = load_model(f'model_{model_idx}.h5')
    S_i = compute_oldens_product([model.layers[l].get_weights()[0] for l in range(n_layers)])
    S_i_norm = S_i / background_threshold_percentile_97_5
    S_i_clipped = np.clip(S_i_norm, -1, 1)
    aggregated_S += S_i_clipped / 100.0
np.savetxt('microbe_metabolite_attribution_matrix.csv', aggregated_S, delimiter=',')

Evaluation signals

• All 100 per-model attribution matrices S_i are computed without NaN or Inf values; dimensions are (n_microbes, n_metabolites).
• Aggregated attribution score matrix has mean ≈ 0 and standard deviation > 0 (indicating meaningful variation across microbe-metabolite pairs).
• After clipping, 100% of values fall within [−1, 1]; no values outside this range remain.
• Rows (microbes) with zero significant attribution scores to any well-predicted metabolite are successfully filtered out.
• 97.5th percentile threshold computed from shuffled background distribution is strictly higher in absolute value than 95% of observed attribution scores (background distribution has expected tail behavior).

Limitations

• Olden's method assumes a multiplicative chain of weight matrices; it may not capture non-linear feature interactions beyond the weight product (mechanistic knowledge is not incorporated).
• Aggregation masks single-model artifacts but does not guarantee biological validity; attribution scores are learned correlations only, not causal mechanisms.
• Background distribution generation via shuffling requires multiple iterations of 10-fold CV on permuted data, which is computationally expensive and must be re-run if data or CV parameters change.
• Normalized thresholds (97.5th percentile for significance, 95th for well-predicted metabolites) are data-dependent; transfer to external cohorts may require re-tuning.
• Not all metabolites may be associated with microbes, resulting in lower prediction correlations and higher thresholds in some datasets (e.g., soil data showed 0.410 vs. 0.129 for other cohorts).

Evidence

• [other] Load trained neural network models from 10 iterations of 10-fold cross-validation (100 total models): "Load trained neural network models from 10 iterations of 10-fold cross-validation (100 total models)."
• [other] Apply Olden's method by computing the product of weight matrices across all layers: "Apply Olden's method by computing the product of weight matrices across all layers: S = ∏(W_l) for l ∈ L, where each row represents a microbe input feature and each column represents a metabolite"
• [methods] Normalize values in each feature attribution score matrix by dividing the significant threshold score identified from the background and clipped values to be between -1 and 1: "We normalized the values in each feature attribution score matrix Si by dividing the significant threshold score identified from the background and clipped values to be between -1 and 1"
• [other] Positive scores indicate increased microbe abundance leads to increased metabolite abundance, and negative scores indicate increased microbe abundance leads to decreased metabolite abundance: "record element-wise values where positive scores indicate increased microbe abundance leads to increased metabolite abundance, and negative scores indicate increased microbe abundance leads to"
• [methods] Any feature attribution score in the observed dataset with an absolute value above the threshold was considered significant: "a threshold was set at the 97.5 percentile. Any feature attribution score in the observed dataset with an absolute value above the threshold was considered significant"
• [results] MiMeNet constructs a score matrix of microbe-metabolite feature attributions using learned network weights obtained from cross-validation training: "using the learned network weights obtained from cross-validation training, MiMeNet constructs a score matrix of microbe-metabolite feature attributions between the microbes and well-predicted"
• [results] Background distribution of SCCs generated through multiple iterations of shuffling the dataset and performing cross-validated evaluation on the shuffled set: "MiMeNet then generates a background distribution of SCCs through multiple iterations of shuffling the dataset and performing a cross-validated evaluation on the shuffled set"