agentsclimarketplace

Optimize

Skill avnath13/evalpilot/skills/optimize

Agent evals on autopilot: find quality bugs in your AI agent, ship a targeted fix, and prove it on a held-out set. Zero-dependency Agent Skill + CLI.

Install
npx -y skills add avnath13/evalpilot --skill optimize

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 20 days oldThe repository was created 20 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Take the top failure cluster, locate the responsible prompt/code/tool, propose a targeted fix (or a constitutional self-critique revision), apply it on a branch, and re-run to prove or disprove the improvement with a before/after comparison. Use when the user wants to fix a known failure mode and verify it. Do NOT use before error analysis (use analyze-failures) and do NOT build a judge for a code-checkable failure, fix it directly (fix-before-eval). Never claim success without the A/B re-run.

SKILL.md

2.0 KB, as published. Nobody here has run it

optimize: targeted fix + A/B proof

Output: a branch/patch (or PR) + compare.md with before/after per target metric.

Procedure

  1. Read the top cluster from .evalpilot/clusters.json, its hypothesis and fix_kind.
  2. Fix-before-eval gate. If fix_kind is code/prompt/tool, fix the cause directly , do NOT build a new judge for a failure a direct edit resolves.
  3. Locate the cause (Grep/Glob): the system prompt, a tool definition, retrieval config, or output-parsing code implicated by the evidence.
  4. Propose ONE targeted change (smallest edit). Two strategies:
    • Targeted edit, a prompt guard, tool-result check, or retrieval tweak.
    • Constitutional self-critique, add a principle the agent checks its own output against before responding (good for subjective/safety failures).
  5. Apply per loop.apply_fixes: propose → branch eval/fix-<cluster> + summary; auto → commit; off → describe only.
  6. Prove it on TEST. Re-run inference + grade on the certify-only test set (and a regression sample of previously-passing cases). Write compare.md.
  7. Hand control back to the orchestrator with the new numbers.

Guardrails

  • NEVER report success without the re-run. A proposed fix is not a proven fix.
  • Always check for regressions on cases that were passing. Report any you introduce.
  • One cluster per iteration, keeps cause and effect legible.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.