Prompt evolution

Two related jobs, in order:

1. Build a judge — a binary pass/fail evaluator for one narrow failure mode. Useful on its own for scoring or grading any AI output.
2. Evolve a prompt — generate variants, score them with that judge, keep the fittest, mutate and recombine, repeat.

Stop after step 1 if the user only wants an eval.

Building a judge

Core principles

1. Binary pass/fail only — no 1-5 scales. Use multiple independent binary checks for granularity.
2. Every judgment needs a critique — explain WHY with specific evidence.
3. Narrow scope per judge — one specific failure mode each. Not "is this good?" but "does this hallucinate facts not in the source material?"
4. Code checks first — if expressible as if/else, use code. Reserve LLM judges for subjective qualities.
5. Criteria drift is normal — draft criteria, review outputs, revise. Iterate.

1. Define the criterion

One narrow, specific failure mode:

• Bad: "Is the response helpful?"
• Good: "Does the response answer the user's specific question without introducing information not present in the provided context?"

2. Write the judge prompt

Send the judge prompt via the discussion-partners skill. It fans the same prompt across multiple models and synthesizes — that's the same multi-model judging pattern this needs.

For single-model judging on a tight budget, invoke one provider skill directly (gemini for batch, codex for high-stakes).

3. Calibrate with examples

Include 2-3 examples (both pass and fail) in the judge prompt. Draw from real outputs, not synthetic ones.

4. Validate alignment

Run the judge on a sample where you know the correct answer. Check:

• True positive rate (catches real failures)
• True negative rate (doesn't flag good outputs)

Target >90% on both. If not, refine the criterion or add examples.

The loop

1. Setup

Define before starting:

• Seed prompt: the starting prompt to optimize
• Test inputs: 3-5 representative inputs the prompt must handle
• Judge criterion: a specific, binary pass/fail criterion (see Building a judge)
• Generations: how many rounds to run (default: 3)
• Population size: variants per generation (default: 4)

2. Generate variants

Launch parallel sub-agents, each using discussion-partners to produce one variant. Apply two genetic operators:

Mutation (always): Literally change words in the prompt. Swap synonyms, delete filler, add specificity, reorder sentences, strengthen or weaken qualifiers. Each mutation should change 1-5 words or phrases — small, targeted edits, not wholesale rewrites.

Here is a prompt: [SEED]. Mutate it: change 1-5 specific words or phrases to potentially improve [CRITERION]. Keep the core intent. Return ONLY the new prompt text.

Crossover (generation 2+): Literally splice sections from two parent prompts. Split each parent into numbered sections (by paragraph or logical block), then take section 1 from parent A, section 2 from parent B, etc. — producing a child that inherits concrete text from both parents.

Here are two prompts that scored well: Parent A: [PROMPT_A] Parent B: [PROMPT_B] Split each into sections. Combine: take some sections from A and some from B to create a child prompt. Then apply 1-3 word-level mutations. Return ONLY the new prompt text.

For generation 1, all variants mutate from the seed. For later generations, half the population uses crossover from the top 2 parents, half uses mutation from a random survivor.

Reflexion pass (before scoring): After generating each variant, critically review it against the archive of prior variants before sending it to the judge. Ask: (1) Is this actually different from what we've already tried? (2) Are there obvious mistakes? (3) Can it be improved without changing the design? Discard or revise duplicates. This prevents wasting evaluation budget on near-copies. (From ADAS two-pass reflexion.)

2b. Feed history as context

When generating variants in generation 2+, include the full history of all prior variants and their scores (survivors AND dead variants) as context to the generator. The generator should see what worked, what failed, and why — so it can explicitly reason about what to try next rather than blindly mutating. The archive IS the prompt engineering; accumulating it gives the generator more signal over time.

3. Score each variant

For each variant, for each test input:

1. Run the variant prompt against a test model (via discussion-partners)
2. Score the output with the judge and its defined criterion
3. Record pass/fail per input

Variant fitness = pass rate across all test inputs.

4. Select

Keep the top 50% of variants (minimum 2). These become parents for the next generation.

Parent selection for next generation: Don't always mutate from the top scorer. Penalize parents that already have many children — a parent used 3 times should be less likely to be picked than an equally-scored parent used once. This prevents over-exploiting one lineage and encourages exploring diverse approaches. (From DGM score-child proportional selection.)

5. Repeat

Mutate and recombine the surviving variants to fill the population. Run for the specified number of generations.

6. Return the winner

Output the highest-fitness prompt with its score and the specific test results.

Example

Seed: "Summarize this article in 3 bullet points"
Criterion: "Each bullet captures a distinct key point, no redundancy, no hallucinated claims"
Test inputs: [3 different articles]
Generations: 3, Population: 4

Generation 1: 4 variants of the seed, scored → keep top 2 Generation 2: 2 survivors + 2 offspring (mutated/recombined), scored → keep top 2 Generation 3: same → return winner

Sub-Agent Orchestration

Each scoring round involves population × test_inputs parallel sub-agent calls. Structure:

1. Variant generation: parallel sub-agents, each uses discussion-partners to create one variant
2. Output generation: parallel sub-agents, each runs one variant × one test input via discussion-partners
3. Judging: parallel sub-agents, each scores one output with the judge

Maximize parallelism at each step. Wait for all results before proceeding to selection.

When to Stop Early

• If the top variant achieves 100% pass rate for 2 consecutive generations
• If fitness plateaus (no improvement for 2 generations)

Archive Persistence

After evolution completes, persist the winning prompt and runner-up to the wiki. This lets future runs seed from proven prompts instead of starting from scratch — the single biggest improvement from meta-learning research (ADAS, Clune lab).

Save with push_files to wiki/notes/Prompt Archive - <Task Name>.md in zachmayer/wiki (Wiki Location), linked from the wiki/hubs/Prompts.md hub — note and hub link in the same commit. The wiki's layout check rejects new top-level folders, so archives are atomic notes like everything else:

# <Task Name>

Source: prompt-evolution run
Date: <ISO date>

## Best Prompt (score: <fitness>)
<the winning prompt text>

## Runner-Up (score: <fitness>)
<second-best prompt text>

## Metadata
- Criterion: <judge criterion used>
- Generations: <N>
- Test inputs: <count>

Before starting a new evolution run, check the archive for an existing entry matching the task — read the Prompts hub with get_file_contents, then the note it links. If found, use the archived best prompt as the seed instead of the user's naive seed. Tell the user: "Found archived prompt for this task (score: X). Using it as seed."

This creates a ratchet: each evolution run builds on the best result from prior runs rather than re-evolving from scratch.