Simulation parameter variation

Simulation Parameter Variation

Summary

Systematically vary one or more input parameters across a defined range in a computational simulation to assess how changes in those parameters affect statistical outcomes and false-positive rates. This skill is essential for validating pathway analysis methods under realistic conditions of incomplete metabolite detection.

When to use

When you have a computational simulation framework (e.g., for ORA p-value distributions) and need to understand how a key experimental constraint—such as metabolite detection coverage (10–100% of the pathway database)—influences statistical validity, false-positive rates, or effect size distributions. Use this skill to isolate the impact of a single variable while holding other conditions constant.

When NOT to use

• You have only a single, fixed parameter configuration and no variation is needed (no sensitivity analysis question).
• The simulation framework is not reproducible or not parameterizable (e.g., closed-source tool with no exposed knobs).
• You are analyzing observed experimental data rather than running controlled simulations—use empirical statistical methods instead.

Inputs

• Jupyter notebook or Python script with parameterizable simulation code
• Parameter range specification (e.g., coverage values: 10, 20, ..., 100%)
• Pathway database or simulated metabolite set definitions
• Statistical test implementation (e.g., ORA function)

Outputs

• Summary statistics table (columns: parameter value, mean p-value, median p-value, false-positive rate, confidence intervals)
• Line or scatter plot: false-positive rate vs. parameter value with error bands
• Boxplots or violin plots: outcome (p-value) distributions across parameter levels
• Raw simulation results per parameter iteration (for reanalysis)

How to apply

Load the simulation framework (typically a Jupyter notebook with reproducible code). Define a range for the parameter of interest (e.g., coverage fraction from 10% to 100% in 10% increments). For each parameter value, execute the full simulation workflow (e.g., run ORA on simulated metabolite sets), record the primary outcome metrics (p-value distributions, false-positive count at p < 0.05 threshold), and store results with associated parameter values. Aggregate results into a summary statistics table with columns for parameter value, mean/median outcome, false-positive rate, and confidence intervals. Visualize the relationship using line plots or scatter plots with error bands for the primary metric (e.g., false-positive rate) as a function of the varied parameter, and produce boxplots or violin plots for outcome distributions across parameter levels. Rationale: varying a single parameter reveals how realistic constraints affect method validity without confounding multiple factors.

Related tools

• Python (Language for implementing and executing parameterized simulations with loops over parameter ranges)
• Jupyter (Interactive notebook environment for running, documenting, and visualizing simulation sweeps and aggregating results)
• metabolomics-ORA (Reference implementation of ORA simulation framework with reproducible code for varying metabolite coverage) — https://github.com/cwieder/metabolomics-ORA.git

Examples

for coverage in range(10, 101, 10):
    results = run_ora_simulation(coverage_fraction=coverage/100, n_replicates=1000)
    summary.append({'coverage': coverage, 'fp_rate': sum(results['p'] < 0.05) / len(results), 'mean_p': results['p'].mean()})
df = pd.DataFrame(summary)
df.to_csv('coverage_sweep_results.csv', index=False)

Evaluation signals

• The summary statistics table is complete with no missing parameter values in the specified range.
• False-positive rate (or other outcome metric) shows a monotonic or expected trend as the parameter varies, consistent with the biological/statistical hypothesis (e.g., false-positive rate decreases with increasing coverage).
• Error bands (confidence intervals) on visualizations are appropriately sized and do not contain contradictory or zero-width intervals.
• Boxplot or violin plot shows non-overlapping median/distribution shifts across parameter levels, confirming parameter effect is detectable.
• All simulation runs are recorded and reproducible from the same code and seed; re-running the workflow with identical parameters yields identical results.

Limitations

• Parameter variation explores only one or two dimensions at a time; interactions between multiple parameters are not captured without factorial design.
• Simulation outcomes depend critically on the quality and realism of the underlying simulation model (e.g., how well the ORA null distribution is specified); parameter variation cannot compensate for model misspecification.
• Computational cost scales linearly or worse with the number of parameter values and simulation replicates; very fine-grained sweeps may be prohibitively expensive.
• False-positive rate estimates depend on the chosen threshold (e.g., p < 0.05); shifting the threshold alters conclusions and must be pre-specified or reported as a sensitivity analysis.

Evidence

• [other] Execute the simulation workflow varying the fraction of detected metabolites across a range of coverage values (e.g., 10–100% of pathway database): "Execute the simulation workflow varying the fraction of detected metabolites across a range of coverage values (e.g., 10–100% of pathway database)"
• [other] For each coverage level, run ORA on simulated metabolite sets and record the distribution of p-values and count false positives (p < 0.05 threshold): "For each coverage level, run ORA on simulated metabolite sets and record the distribution of p-values and count false positives (p < 0.05 threshold)"
• [other] Aggregate results into a summary statistics table with columns for coverage percentage, mean/median ORA p-value, false-positive rate, and confidence intervals: "Aggregate results into a summary statistics table with columns for coverage percentage, mean/median ORA p-value, false-positive rate, and confidence intervals"
• [other] Generate a line or scatter plot showing false-positive rate as a function of coverage with error bands, and produce boxplots or violin plots of p-value distributions across coverage levels: "Generate a line or scatter plot showing false-positive rate as a function of coverage with error bands, and produce boxplots or violin plots of p-value distributions across coverage levels"
• [intro] The Python code to generate the results is contained within the Jupyter notebook: "The Python code to generate the results is contained within the Jupyter notebook"