Harness optimization loop

Harness Optimization Loop

Overview

Improve the system around the model, not the model weights: prompts, tool order, context selection, verification gates, routing, memory use, and workflow shape. Optimize only when there is an evaluation surface.

When to Use

Use this skill:

• A prompt, workflow, or tool sequence repeatedly underperforms.
• A task needs a more reliable agent harness before scaling or reuse.
• There is a benchmark, fixture set, CI check, golden output, human rubric, or repeatable manual test.
• The user asks to optimize an agent workflow, prompt, harness, evaluator, or multi-agent process.

Do not use it when there is no way to compare candidates, when a single obvious bug fix is available, or when the change would create broad platform complexity for a narrow one-off task.

Workflow

1. Define the target behavior.

   • What should improve?
   • What must not regress?
   • What evidence will decide between candidates?

2. Capture the baseline.

   • Save the current prompt, workflow, config, or instructions.
   • Run the existing evaluation or reproduce the failure.
   • Record the result and the exact command or manual check.

3. Isolate variables.

   • Change one dimension per candidate when possible.
   • Separate bug fixes from prompt wording changes.
   • Separate retrieval/context changes from tool-routing changes.

4. Generate small candidates.

   • Prefer 2-4 candidates.
   • Keep each candidate cheap to test and easy to revert.
   • Avoid adding new abstractions unless the evaluation demands them.

5. Evaluate and compare.

   • Use the same inputs and scoring method for every candidate.
   • Record pass/fail, score, cost, latency, and notable failure modes when available.
   • Prefer evidence over intuition.

6. Select, integrate, and verify.

   • Keep the best candidate or combine only independently validated changes.
   • Run the project's normal verification gate.
   • Archive enough notes for the next iteration.

Candidate Log Template

Goal:
Baseline:
Evaluation:

Candidate A:
- Change:
- Result:
- Failure mode:

Candidate B:
- Change:
- Result:
- Failure mode:

Decision:
Follow-up:

What Counts As Harness

• System or developer instructions.
• Prompt templates and examples.
• Retrieval, memory, and context selection.
• Tool choice and call order.
• Subagent routing and review loops.
• Verification gates and scoring scripts.
• File-system trace layout for future iterations.

Common Mistakes

• Optimizing without a metric or repeatable check.
• Changing prompt, tool order, and evaluator at the same time.
• Keeping a candidate because it feels cleaner while the score regresses.
• Summarizing historical traces so aggressively that debugging evidence is lost.
• Treating this as model training. This skill changes the harness only.