agentsclimarketplace

Prompt optimizer

Skill haabe/mycelium/plugins/mycelium/skills/prompt-optimizer

AI made building cheap. It didn't make deciding cheap. Mycelium is a Claude Code harness that makes your agent run discovery and weigh evidence before it writes code. It earns the right to start. Built for software, courses, AI tools, and services.

Install
npx -y skills add haabe/mycelium --skill prompt-optimizer

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

What its author says it does

Copied from the file, not written here

A/B test CLAUDE.md instruction changes against eval benchmarks. Capture baselines, test variants, compare results.

SKILL.md

2.1 KB, as published. Nobody here has run it

Prompt Optimizer

Systematically improve Mycelium instructions through measurement. Adapted from n-trax.

Commands

baseline -- Capture current performance

  1. Run /mycelium:eval-runner run-split optimization — record as optimization scores
  2. Run /mycelium:eval-runner run-split holdout — record as holdout scores
  3. Record both to .claude/optimization/baseline.json: timestamp, CLAUDE.md hash, optimization metrics, holdout metrics, overall and per-category metrics

test <variant> -- Test a variant

  1. Read variant from .claude/optimization/variants/<variant>.md
  2. Apply the CLAUDE.md changes described
  3. Run /mycelium:eval-runner run-split optimization — this is the hill-climbing signal
  4. Run /mycelium:eval-runner run-split holdout — this validates generalization
  5. Store results in .claude/optimization/results/<variant>.json
  6. Compare against baseline. Flag overfitting if optimization improves but holdout degrades.
  7. Do NOT auto-revert -- let user decide

report -- Compare all variants

Generate comparison table with split-aware columns:

| Variant | Opt Pass Rate | Holdout Pass Rate | Delta Opt | Delta Holdout | Overfit? | Decision |

Flag Overfit? = YES when optimization delta is positive but holdout delta is negative.

exemplar <eval-name> -- Capture winning trajectory

After a clean eval win (1 iteration, fast), save the approach to .claude/optimization/exemplars/.

Workflow

  1. Capture baseline
  2. Hypothesize an instruction improvement
  3. Document in variants/ directory
  4. Test the variant
  5. Compare via report
  6. Keep or revert based on data
  7. Capture exemplars from clean wins

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.