Parameter optimization

Parameter Optimization

Goal

Provide a workflow to design experiments, rank parameter influence, and select optimization strategies for materials simulation calibration.

Requirements

• Python 3.10+
• No external dependencies (uses Python standard library only)

Inputs to Gather

Before running any scripts, collect from the user:

Input	Description	Example
Parameter bounds	Min/max for each parameter with units	kappa: [0.1, 10.0] W/mK
Evaluation budget	Max number of simulations allowed	50 runs
Noise level	Stochasticity of simulation outputs	low, medium, high
Constraints	Feasibility rules or forbidden regions	kappa + mobility < 5

Decision Guidance

Choosing a DOE Method

Is dimension <= 3 AND full coverage needed?
├── YES → Use factorial
└── NO → Is sensitivity analysis the goal?
    ├── YES → Use quasi-random (preferred; "sobol" is accepted but deprecated)
    └── NO → Use lhs (Latin Hypercube)

Method	Best For	Avoid When
lhs	General exploration, moderate dimensions (3-20)	Need exact grid coverage
quasi-random	Sensitivity analysis, uniform coverage (preferred)	Very high dimensions (>20)
sobol	Deprecated alias of quasi-random (emits a warning)	New code (use quasi-random)
factorial	Low dimension (<4), need all corners	High dimension (exponential growth)

Factorial sizing: the factorial grid is levels evenly spaced values per parameter, producing exactly levels ** params samples. Set the resolution explicitly with --levels (e.g. --params 2 --levels 4 -> 16 samples). If you use --budget instead, the script back-computes levels = round(budget ** (1/params)) and warns whenever the realized sample count differs from the requested budget (e.g. --budget 20 --params 2 realizes 16 samples). For an exact design, pass a perfect power (--budget 16) or, preferably, --levels.

Choosing an Optimizer

Is dimension <= 10 AND budget <= 100?
├── YES → Bayesian Optimization
└── NO → Is dimension <= 20?
    ├── YES → CMA-ES
    └── NO → Random Search with screening

Noise Level	Recommendation
Low	Gradient-based if derivatives available, else Bayesian Optimization
Medium	Bayesian Optimization with noise model
High	Evolutionary algorithms or robust Bayesian Optimization

Script Outputs (JSON Fields)

Script	Output Fields
scripts/doe_generator.py	samples, method, coverage (count, dimension; plus levels and a top-level requested_budget/note for factorial)
scripts/optimizer_selector.py	recommended, expected_evals, notes
scripts/sensitivity_summary.py	ranking, notes
scripts/surrogate_builder.py	model_type, metrics (mse, cv_error, output_variance), notes

Workflow

1. Generate DOE with scripts/doe_generator.py
2. Run simulations at DOE sample points (user's responsibility)
3. Summarize sensitivity with scripts/sensitivity_summary.py
4. Choose optimizer using scripts/optimizer_selector.py
5. (Optional) Fit surrogate with scripts/surrogate_builder.py

CLI Examples

# Generate 20 LHS samples for 3 parameters
python3 scripts/doe_generator.py --params 3 --budget 20 --method lhs --json

# Full factorial with 4 levels per parameter (2 params -> 16 samples)
python3 scripts/doe_generator.py --params 2 --levels 4 --method factorial --json

# Rank parameters by sensitivity scores
python3 scripts/sensitivity_summary.py --scores 0.2,0.5,0.3 --names kappa,mobility,W --json

# Get optimizer recommendation for 3D problem with 50 eval budget
python3 scripts/optimizer_selector.py --dim 3 --budget 50 --noise low --json

# Build surrogate model from simulation data
python3 scripts/surrogate_builder.py --x 0,1,2 --y 10,12,15 --model rbf --json

Conversational Workflow Example

User: I need to calibrate thermal conductivity and diffusivity for my FEM simulation. I can run about 30 simulations.

Agent workflow:

1. Identify 2 parameters → --params 2
2. Budget is 30 → --budget 30
3. Use LHS for general exploration:

   python3 scripts/doe_generator.py --params 2 --budget 30 --method lhs --json
   

4. After user runs simulations and provides outputs, summarize sensitivity:

   python3 scripts/sensitivity_summary.py --scores 0.7,0.3 --names conductivity,diffusivity --json
   

5. Recommend optimizer:

   python3 scripts/optimizer_selector.py --dim 2 --budget 30 --noise low --json
   

Error Handling

Error	Cause	Resolution
params must be positive	Zero or negative dimension	Ask user for valid parameter count
budget must be positive	Zero or negative budget	Ask user for realistic simulation budget
argument --method: invalid choice: <value> (choose from lhs, sobol, quasi-random, factorial)	Invalid method (argparse)	Use decision guidance to pick a valid method
could not convert string to float: <token>	Non-numeric value in --scores/--x/--y	Reformat as 0.1,0.2,0.3
scores must be a comma-separated list	Empty --scores input	Provide at least one numeric score

Verification checklist

• Recorded the exact doe_generator.py coverage.count and confirmed it matches the intended design — for factorial, verified count == levels ** params and that no note/requested_budget mismatch warning was emitted (or that the realized count is acceptable).
• Confirmed the chosen --method matches the Decision Guidance for the actual dimension/goal, and that quasi-random was used instead of the deprecated sobol alias (no DeprecationWarning in output).
• Recorded the optimizer_selector.py recommended strategy and expected_evals, and verified expected_evals <= budget so the plan is feasible within the stated evaluation budget.
• Logged the sensitivity_summary.py ranking and checked whether the top sensitivity is < 0.1 (the "All sensitivities are low" note); if so, did not over-interpret the ranking and revisited the output metric.
• For surrogate fits, judged quality with metrics.cv_error (leave-one-out), NOT in-sample mse — especially for rbf, where mse is near zero by construction — and compared cv_error against metrics.output_variance to confirm the surrogate beats the constant-mean baseline.
• Confirmed any reported cv_error is a finite number (not NaN), i.e. there were enough samples for leave-one-out (poly: n > degree+1; rbf: n >= 3).

Common pitfalls & rationalizations

Tempting shortcut	Why it's wrong / what to do
"RBF surrogate mse is ~0, so the model is excellent."	RBF is an exact interpolant — in-sample mse is near zero by construction and says nothing about generalization. Judge fit with metrics.cv_error and compare it to output_variance.
"I asked for --budget 20 factorial, so I got 20 samples."	Factorial honors levels ** params, not the budget; --budget 20 --params 2 realizes 16 samples and emits a note/warning. Use --levels for an exact, intended design.
"sobol gives me a true Sobol low-discrepancy sequence."	sobol is a deprecated alias that emits a DeprecationWarning and uses a simplified golden-ratio additive recurrence, not a true Sobol sequence. Use quasi-random; for production Sobol use scipy.stats.qmc.
"The optimizer recommendation is just advice — budget doesn't matter."	The recommendation is gated on dimension AND budget (BO only for dim<=10 AND budget<=100), and expected_evals is capped at the budget. Record both and confirm the plan fits the real budget.
"One sensitivity score is highest, so that parameter dominates."	The script only sorts the scores you pass in; it computes no sensitivity itself. If the top score is < 0.1 it flags that all sensitivities are low — get the scores from a real screening/Sobol analysis before trusting the ranking.
"It printed JSON without erroring, so the result is valid."	Exit success only means inputs parsed. Verify the design size, expected_evals <= budget, a finite cv_error, and that the surrogate beats output_variance before trusting any output.

Security

Input Validation

• sensitivity_summary.py validates --names against [a-zA-Z_][a-zA-Z0-9_ .-]* with a 200-char limit, preventing shell metacharacter injection via crafted parameter names
• All numeric list inputs are validated as finite numbers (NaN/Inf rejected)
• Comma-separated value lists are capped (10,000 for scores, 100,000 for surrogate data) to prevent resource exhaustion
• doe_generator.py caps dimension at 1,000 and budget at 1,000,000; optimizer_selector.py caps dimension at 100,000 and budget at 10,000,000
• --method is validated against a fixed allowlist (lhs, quasi-random/sobol, factorial); sobol is an accepted but deprecated alias of quasi-random
• --noise is validated against a fixed allowlist (low, medium, high)
• --model (surrogate type) is validated against a fixed allowlist (rbf, poly)
• --levels (factorial grid resolution) is validated as an integer in [2, 1000]

File Access

• Scripts read no external files; all inputs are provided via CLI arguments
• Scripts write only to stdout (JSON output); no files are created unless the agent explicitly uses the Write tool

Tool Restrictions

• Read: Used to inspect script source, references, and user data files
• Write: Used to save DOE sample plans, sensitivity rankings, or optimizer recommendations; writes are scoped to the user's working directory
• Grep/Glob: Used to locate relevant files and search references
• The skill's allowed-tools excludes Bash to prevent the agent from executing arbitrary commands when processing user-provided parameter names and constraints

Safety Measures

• No eval(), exec(), or dynamic code generation
• All subprocess calls use explicit argument lists (no shell=True)
• Reduced tool surface (no Bash) limits the agent to read/write operations only
• Parameter names are sanitized before use, preventing injection via crafted identifiers

Limitations

• Not for real-time optimization: Scripts provide recommendations, not live optimization loops
• Surrogate is lightweight: surrogate_builder.py fits a real 1-D least-squares polynomial (poly) or Gaussian RBF interpolant (rbf) using only the standard library and reports honest residual mse, leave-one-out cv_error, and the data output_variance; for production use scipy/scikit-learn/GPyTorch. For rbf, in-sample mse is near zero by construction (exact interpolation) — judge fit quality with cv_error
• No automatic simulation execution: User must run simulations externally and provide results

References

• references/doe_methods.md - Detailed DOE method comparison
• references/optimizer_selection.md - Optimizer algorithm details
• references/sensitivity_guidelines.md - Sensitivity analysis interpretation
• references/surrogate_guidelines.md - Surrogate model selection

Version History

• v1.2.2 (2026-06-24): Added Verification checklist and Common pitfalls & rationalizations sections to drive evidence-based use of the DOE, optimizer, sensitivity, and surrogate scripts
• v1.2.0 (2026-06-23): Real surrogate fits (poly least-squares, rbf interpolation) with honest mse/cv_error/output_variance; explicit factorial --levels with budget-mismatch warnings; BO dimension cutoff harmonized to dim<=10; corrected Security/Error-Handling/output-field docs to match script behavior
• v1.1.0 (2024-12-24): Enhanced documentation, decision guidance, conversational examples
• v1.0.0: Initial release with core scripts