Skill improver

Skill Improver — Autoresearch for SKILL.md

Core Philosophy: The human programs the researcher, not the research. Apply Karpathy's autoresearch methodology — greedy hill climbing with keep/discard against a scalar metric — to autonomously improve Claude Code skills.

Invocation

Argument grammar:

/skill-improver <mode> <target> [--opts]

• <mode> — improve (default) | score | freshen | trigger | philosophy | batch
• <target> — skill name (e.g. gh-cli), absolute SKILL.md path, --all, or glob (e.g. vllm-*)
• [--opts] — mode-specific flags (e.g. --iterations 15, --probe-budget 30, --runs-per-query 5)

Examples:

/skill-improver freshen autoresearch
/skill-improver score gh-cli
/skill-improver improve ~/.claude/skills/helm
/skill-improver trigger vllm-caching
/skill-improver trigger gh-cli --missed "find issue with label X"
/skill-improver batch freshen --all
/skill-improver freshen --group 'vllm-*'

If <mode> is omitted, default to improve. If <target> is omitted and mode is not batch, prompt the user. For batch, the target after batch selects the sub-mode (freshen, improve, trigger, or philosophy, default improve); the target list comes from scripts/scan-skills.sh. The --missed "<phrase>" flag (trigger mode only, repeatable) seeds the eval set with user-reported failures as gold should-trigger queries.

The Improvement Loop

Phase 0: Setup

1. Identify the target skill. Accept a path, or run scripts/scan-skills.sh (or Glob pattern **/SKILL.md under ~/.claude/skills/ and .claude/skills/) to list candidates. Do NOT search ~/.claude/plugins/ — those are managed externally.
2. Read the target skill's entire directory: SKILL.md and any references/, examples/, scripts/, assets/ present.
3. Read <skill>/references/improvement-backlog.md if it exists. This file carries open issues from prior skill-improver runs — ceiling-hit items that require multi-file restructure or author judgment. Do NOT re-propose items already listed there unless new evidence (e.g. the ceiling is now breakable in one iteration due to earlier structural changes). Items resolved mid-loop get moved to the backlog's "Resolved this pass" section in Phase 6.
4. Read both references/quality-rubric.md (scoring criteria) and references/improvement-patterns.md (concrete before/after patterns by dimension) from the skill-improver directory. Both are non-optional — skipping the patterns file means later iterations propose changes that miss documented techniques (Pattern 8.2 terminology standardisation, Pattern 6.1 redundancy removal, Pattern 9.3 frontmatter fields). Feeling unsure what to try next at any phase is a symptom of skipping this read. Apply the Boris Alignment Check (rubric §"Boris Alignment Check") on the baseline — three diagnostic patterns (strict workflow scaffolding, up-front context dumps, model-version compensation) cap Dims 6, 4, and 9 respectively. Caps surface as cross-cutting structural issues that should be lifted ahead of cosmetic dim improvements of the same magnitude.
5. Establish a baseline score by evaluating the skill against the rubric.
6. Spawn a blind scoring agent on the baseline (see "Blind Validation" section). First snapshot the skill: cp -a <skill-dir> /tmp/<skill-name>-baseline. Then run the agent in the background while the loop proceeds. This is non-optional. The baseline blind agent is the only check on Phase 1's self-score — without it the entire run rests on whatever bias the loop's self-scoring carries. If the runtime cannot spawn agents, run the same prompt manually in a fresh session and paste back the result before entering Phase 2. Do NOT proceed to Phase 6 without both a baseline AND a final blind score on record.
7. Initialize a results log (in-memory or scratch file) with header: iteration | score | delta | status | description.
8. Log iteration 0 as baseline.

Phase 1: Evaluate (Score the Skill)

Score the skill on 10 dimensions (each 0–10, summed to 0–100) using the detailed criteria and scoring template in references/quality-rubric.md (loaded in Phase 0).

Cold-score discipline. When scoring at any phase, read the current file fresh and assign each dimension against the rubric criteria with no reference to prior iteration scores. Do NOT compute the new score by adding deltas to the old. Delta math hides regressions in dimensions not being watched.

Phase 2: Hypothesize (Pick One Improvement)

Identify the single lowest-scoring dimension (or highest-impact if tied). If the baseline blind agent has returned with flagged dimensions (2+ gap), use the agent's specific justification text — not just the number — to inform the hypothesis. Formulate one specific change:

• What to change and why
• Expected score impact
• Complexity cost (lines added/removed, new files)

Consult references/improvement-patterns.md for concrete before/after patterns organized by dimension.

Check the rejected-edit buffer first. The run log's discard rows (Phase 5 requires them to carry shape + reason) are this run's rejected-edit buffer. Do NOT re-propose an edit of the same shape against the same section that a prior iteration discarded — change the dimension, the section, or the mechanism. The only exemption: a change kept since the discard has plausibly removed the reason it failed; if claiming that, name the kept iteration and the removed reason in the hypothesis.

Factual-claim hypotheses require a probe. If the change would alter a version, date, model name, API, flag, or any other external-world claim, run an online verification BEFORE mutating (see Operating Rules §"The Skill Outranks Training Data") — the claim is likely newer than the model's knowledge cutoff, and "fixing" it from memory regresses the skill.

The simplicity criterion (from autoresearch): A small improvement that adds ugly complexity is not worth it. Removing something and getting equal or better results is a great outcome. A +1 score that adds 20 lines of noise? Skip. A +1 from deleting redundant content? Keep.

Format-only hypotheses are low expected value. SkillLens (arXiv:2605.23899) measured skill format (ordered list vs prose vs checklist vs unordered) as statistically non-significant on every tested target, while changing what the skill says was significant on 5/6 — prefer content hypotheses (mechanism + remedy, blacklists, coverage) over reformatting, renaming, or restructuring-for-looks. (The 2026-07-18 self-run confirmed this empirically: all three format/naming iterations discarded at Δ0.)

Phase 3: Mutate (Make the Change)

Apply exactly one change per iteration, diff minimal. Do NOT bundle multiple improvements — bundling attributes the score lift to the wrong cause, so the next loop pivots to the wrong category.

Phase 4: Re-evaluate (Score Again)

1. Re-score the skill using the same rubric.
2. Compare to previous best score.

Decision rule:

• Score improved by +2 or more → KEEP. Log as keep. This is the new baseline. On every keep: commit (per §Git as State Machine), or — when commits are not permitted — snapshot the kept file to the scratch directory. Anomaly gate (+5 or more): A single change that lifts the total by +5 or more is presumed inflated until proven otherwise. Do NOT rationalize the deltas. Instead: open the rubric fresh, read the current file as if it were new, and score each dimension cold. If the cold total differs from the delta-math total by 2 or more in either direction, the cold score wins. Most +5 jumps shrink to +3 under cold rescore — that is the finding, not a failure of the change. Log both totals in the iteration row.
• Score improved by exactly +1 (noise zone) → a bare +1 is inside self-scoring noise — cold rescores routinely move a total by ±1–2, so the +1 may be the scorer, not the change. If the change also simplifies (net lines removed), KEEP as keep (simplification). Otherwise cold-score the affected dimension(s) fresh; KEEP only if the +1 reproduces, else revert and log as discard (noise). Noise discards count toward the ceiling-mapped stop condition like any other discard.
• Score equal, but simpler → KEEP. Log as keep (simplification).
• Score equal or worse → DISCARD. Revert via git checkout -- <file> ONLY if every prior keep is committed; with uncommitted keeps, restore the last kept snapshot instead — a whole-file checkout reverts to git HEAD and silently destroys them. (Not git-tracked: undo the edit.) Log as discard. The discard row must name WHAT was tried (change shape + target section) and WHY it failed — discard rows are the rejected-edit buffer Phase 2 consults.
• Change broke something → REVERT. Log as crash. Fix and continue.

Phase 5: Log and Loop

1. Append result to the log: iteration | score | delta | status | description. Use a single declared score column for trend math — pick self OR blind and stay with it across iterations. Do NOT mix self-scores and blind-scores in the same delta column to make iterations look bigger; if both are tracked, log them as separate columns side by side and compute deltas within each column.
2. Print a one-line status, e.g.: [iter 3] score: 74 (+2) — keep — moved API docs to references/api.md.
3. Go to Phase 2 and pick the next improvement.

Reflect (every 5 iterations): Categorize all iterations by type (simplification, style fix, restructuring, content addition, trigger tuning). If the last 5 were all the same category, force the next hypothesis to be a different category. Print: [reflect] N kept from <category>, pivoting to <new category>

Stop conditions:

• Score reaches 90+ AND no dimension is below 7.
• Ceiling mapped: 5+ consecutive discards spanning at least 2 different improvement categories. This is not failure — it means the skill is near its quality ceiling. Report as a positive finding: which categories were tried, what the ceiling is, and what would require the author's input to break through.
• Structural ceiling claim requires evidence. "Structural ceiling" stops require at least 2 logged discards naming the patterns that were attempted and why each failed. A run with zero discards has not mapped any ceiling — it has stopped early. Reasoning "the next iteration would just be a discard" without actually trying it is the cheat. Try it.
• User interrupts.
• 10 iterations completed (default cap; user can override).

What a stop is NOT:

• Not "+N feels like enough". The metric drives the loop; subjective comfort with the gain does not.
• Not "the score is good and I am tired". Read on.
• Not "Dim X is capped, so further improvement is impossible". Other dims may still be liftable. Stop only when the rubric criteria for stopping match.

On stop: Spawn a final blind scoring agent (see "Blind Validation"). Print both comparison tables (baseline + final) and the overall results summary.

Phase 6: Persist the backlog

Before declaring the run done, update <skill>/references/improvement-backlog.md (create the file if absent). This is non-optional — ceiling findings that exist only in chat disappear when the session ends.

The two sections and their admission rules — what qualifies as Open (attempted, could not be applied in one iteration; not a wishlist), what counts as Resolved this pass (a mutation the metric registered; not a placeholder file), the carry-forward marker, and the append-only rule that preserves prior passes' discard rationales — live in references/backlog-format.md. Read it before writing the file.

---

Operating Rules

Never Stop (Unless Asked)

Run the loop continuously. Do not ask permission between iterations. The user may be away. Print status lines so they can review when they return.

Git as State Machine

When improving skills in a git-tracked directory:

• Commit each kept improvement individually.
• Use git diff to show what changed on discard before reverting.
• The branch tip always represents the best-known version.

Prioritize Deletion Over Addition

In practice, removing redundant content produces the largest per-iteration score gains. When choosing between an additive improvement (+1 from adding content) and a subtractive one (+1 from deleting content), prefer deletion — it improves simplicity as a side effect.

One File at a Time

Each iteration targets one file. If the improvement requires touching multiple files (e.g., moving content from SKILL.md to references/), that counts as one atomic change.

The split test for atomicity. "Atomic" is not a word — it is a constraint. State the change in 10 words, present-tense, single verb. "Move gotchas section to references/gotchas.md." If the honest sentence needs an "and" — "move content to references/ AND fix second-person AND tighten terminology" — it is three iterations, not one. Pure relocation is allowed; relocation that quietly rewrites prose is not. If a structural move starts editing a sentence's wording, stop, finish the move with the prose unchanged, score, then propose the prose edit as the next iteration. The reason: bundled iterations attribute the score lift to the wrong cause, which means future loops will pick the wrong category to pivot to.

Preserve the Author's Intent

The skill reflects the author's domain expertise. Improve structure, clarity, and adherence to best practices. Do NOT rewrite the author's domain knowledge or change what the skill teaches — only how it teaches it.

The Skill Outranks Training Data

Target skills are freshened continuously — their factual claims (versions, release dates, model names, APIs, flags, pinned SHAs) are often NEWER than the model's knowledge cutoff. Treat the skill's existing text as more current than the model's prior, never the reverse. This rule applies in EVERY mode, not just freshen:

• Never mutate an external-world claim from memory. If a hypothesis requires changing one, verify online first (gh / WebFetch / WebSearch, freshen-style probe) and cite the source in the iteration log — or drop the hypothesis. "I know this is wrong" is not evidence; the probe is.
• Downgrade alarm: wanting to lower a version, move a date backward, or revert a claim to an older state is the signature of training-data staleness — the skill was probably freshened past the cutoff. Mandatory online check before touching it; expect to find the skill is right.
• This binds blind scorers too — the validation prompt instructs them to check sources.md stamps instead of scoring Dim 9 down from memory, and the loop must not act on a blind agent's "wrong version" finding without its own probe.

---

Blind Validation

Self-evaluation bias is real — the agent that wrote improvements tends to score them generously. Blind validation uses independent subagents that have never seen the skill to score it objectively. Run it twice: at baseline and after the loop.

When to Run

1. Baseline — after the self-score in Phase 0 step 5, spawn a blind scoring agent in the background. It runs in parallel with the improvement loop.
2. Final — after the loop stops, spawn another blind scoring agent on the final version.

Agent Prompt, Model Pin, and Comparison Table

The scorer prompt (verbatim, substitute paths), the model pin, the opt-in-guarded parallel-scoring variant, and the bias-check table format live in references/blind-validation.md. Read it when spawning either agent. Two rules that bind without reading it: pin the agent explicitly (currently model: "opus" — Opus 5 at xhigh; never inherit the session default), and print the bias-check table after each agent returns, flagging every dimension where self and blind differ by 2 or more.

---

Batch Mode

To improve multiple skills:

1. Run scripts/scan-skills.sh to find all SKILL.md files in scope.
2. Score each skill (baseline only) and print a ranked table.
3. Sort by score ascending (worst first).
4. Run the improvement loop on each, starting from the worst. Cap at 5 iterations per skill in batch mode.
5. Print a final summary table: skill name, baseline score, final score, delta, number of kept changes.

Dynamic workflows (Fable 5 / Opus 5, Claude Code v2.1.154+). Batch mode is multi-agent orchestration — when the user has opted into the Workflow tool, reuse the saved driver scripts/batch-workflow.js (a recon→apply→blind pipeline, median-of-3 final blind): Workflow({scriptPath: "${CLAUDE_SKILL_DIR}/scripts/batch-workflow.js", args: ["keda", "helm", ...]}). args takes bare names, absolute dirs, or {dir, hints} objects. Per-skill loops keep one change per iteration so cause stays attributable; recon and apply agents inherit the session model while the blind scorers carry the same explicit pin as a solo run, and no agent does git ops — commit per-skill after review. Without opt-in, run skills sequentially as above.

Native loops (Claude Code /loop v2.1.71+, /goal v2.1.139+). For recurring or goal-driven runs, drive this skill with the harness's loop primitives: /loop <interval> /skill-improver batch freshen --all for scheduled passes, or /goal with a checkable stop ("every skill scores ≥85, stop after N tries") — /goal's evaluator-checked stop condition maps directly onto this skill's scalar metric. Size batch fan-outs against three live caps: 20 concurrent subagents (v2.1.217 default, CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS), 200 subagents + 200 web searches per session (v2.1.212), and the medium workflow size guideline of <15 agents (v2.1.219 default, workflowSizeGuideline) — a batch wider than these queues silently or trips a guideline warning mid-run. Official guidance: https://claude.com/blog/getting-started-with-loops (2026-06-30).

Standalone Evaluation (No Loop)

When the user only wants a quality score without iterating:

1. Read the target skill and references/quality-rubric.md from the skill-improver directory.
2. Score all 10 dimensions using the scoring template from the rubric.
3. Print the results table. Highlight the lowest dimension and recommend the single highest-impact improvement.
4. If Dim 9 is capped by sources.md staleness (see rubric §Dim 9), recommend running freshen <skill> as the single highest-impact next step.
5. Stop. Do not enter the improvement loop unless asked.

/doctor is the first-party sibling, not a substitute. Anthropic ships claude doctor / /doctor to "rightsize your skills, and CLAUDE.md files" (context-engineering blog, 2026-07-24; bundled skill since v2.1.205). It is a one-shot simplification pass with no metric, no keep/discard, and no blind check — run it first for free hypotheses, then use this skill when the question is did the change measurably help. Do NOT report /doctor output as a score.

Dim 10 is capped until net value is measured. A skill is not neutral-to-positive by default — SkillLens measured 25% of skills as net-harmful against a no-skill baseline (47% in the worst domain). The rubric's §"Negative-Transfer Gate" caps Dim 10 at 8 while unmeasured and at 2 when the skill loses to no-skill. Measuring means delta_pass_rate from skill-creator's aggregate_benchmark, not a judgement about the text.

Scope boundary — every metric here scores the skill's text, never its outputs. SkillLens measured text-only judging at 46.4% accuracy against real utility, so a rising rubric score is not evidence the skill's results improved. To measure that, run the official skill-creator plugin's eval loop (/plugin install skill-creator@claude-plugins-official): assertions in evals/evals.json, one clean-context subagent per case, with-skill vs without-skill benchmark.json, blind A/B between versions. Methodology: https://agentskills.io/skill-creation/evaluating-skills

---

Freshen Mode

Probe a skill's external references for staleness and apply verified updates in place — same keep/discard loop as improve, but hypotheses come from online evidence (release notes, doc commits, deprecation signals), not rubric scores.

Invocation: freshen <skill-path> · --all · --group <glob>. Defaults to apply; for a read-only readout use Standalone Evaluation (Dim 9 tracks sources.md).

Full phase workflow (F0→F6), batch mode, and anti-patterns live in references/freshen-patterns.md §"Freshen Mode Workflow" with the extraction heuristics, probe templates, and classification rules. Read it when running freshen.

---

Trigger Mode

Measure and tune a skill's frontmatter description (and when_to_use) so it fires when it should and stays silent when it shouldn't. Same keep/discard hill-climbing as improve, but the metric is trigger rate against an eval set — the methodology Anthropic's skill-creator uses (60/40 train/test, 7 runs/query for decisions, blinded test scores, ≤1024-char cap).

Use trigger mode when: a user reports "the skill didn't fire" / "Claude isn't using my skill", or a description is too vague, narrow, or keyword-collision-y. Trigger-mode measures Dim 1 empirically via claude -p (scripts/probe-trigger.py).

Full phase workflow (T0 Setup → T7 Apply/persist), batch mode, and anti-patterns live in references/trigger-patterns.md §"Trigger Mode Workflow" with the eval-set construction rules, probe mechanism, and mutation patterns. Read it when running trigger.

---

Philosophy Mode

Cheap weekly check that runs the three Boris-derived signals as one pass without spinning up the full 10-dim rubric or the trigger eval set. Sibling to freshen and trigger. Sourced from Boris Cherny (creator of Claude Code, Anthropic; Lenny's podcast 2026). Output is a Boris score (0-3 anti-patterns flagged) plus the existing dim caps that fire as a side-effect.

Invocation: philosophy <skill-name> · batch philosophy --all. Surfaces findings only — never auto-applies mutations; the operator decides.

Full phase workflow (P0 Setup → P4 Persist), Boris score interpretation, batch leaderboard, and anti-patterns live in references/philosophy-patterns.md. Read it when running philosophy, along with the three check sections P0 uses: quality-rubric.md §"Boris Alignment Check", freshen-patterns.md §"4b. Scaffolding Decay Probes", trigger-patterns.md §"Minimalism test (Boris alignment)".

---

Additional Resources

Reference Files

• references/quality-rubric.md — Full scoring rubric with sub-criteria, examples of each score level, and common failure patterns. Load this before scoring.
• references/improvement-patterns.md — Catalog of common improvements organized by dimension, with before/after examples.
• references/freshen-patterns.md — The full Freshen Mode workflow (F0–F6) plus reference-extraction heuristics, probe templates (gh CLI / WebFetch / WebSearch), and classification rules. Load when running freshen.
• references/trigger-patterns.md — The full Trigger Mode workflow (T0–T7) plus eval-set construction, mutation patterns by failure type, decision rules, and worked example. Load when running trigger.
• references/philosophy-patterns.md — The full Philosophy Mode workflow (P0–P4) plus Boris score interpretation, batch leaderboard, and anti-patterns. Load when running philosophy.
• references/blind-validation.md — The blind-scorer prompt, model pin, parallel-scoring variant, and bias-check table format. Load when spawning a baseline or final blind agent.
• references/backlog-format.md — The Open / Resolved this pass section shapes, admission rules, and append-only history rule. Load when writing a target skill's improvement-backlog.md in Phase 6.
• references/anthropic-skill-design.md — Anthropic's skill design practices, complete frontmatter reference, Agent Skills standard, and platform constraints. Consult when scoring Dimensions 1, 2, 8, and 9.
• references/sources.md — Dated per-URL index of official docs, specs, changelogs, and blog posts. Freshen Mode reads and stamps Last verified: / Pinned: fields here.
• <skill>/references/improvement-backlog.md (per-target, not in skill-improver's own dir) — Carries ceiling findings across skill-improver runs. Read in Phase 0 step 3; updated in Phase 6. Each target skill that has ever been through skill-improver should have one.
• <skill>/references/trigger-evals.json (per-target) — Persistent eval set for Trigger Mode. Built on first trigger run; reused and extended on subsequent runs. Schema: [{"query": str, "should_trigger": bool, "source": str}, ...].

Scripts

• scripts/scan-skills.sh — Find all SKILL.md files in profile and project scopes. Outputs paths sorted by modification time.
• scripts/batch-workflow.js — Reusable Workflow-tool driver for batch improve + freshen (recon → apply → blind pipeline, median-of-3 final blind). Skill list comes from args. Invoke with Workflow({scriptPath: "${CLAUDE_SKILL_DIR}/scripts/batch-workflow.js", args: [...]}). See Batch Mode § Dynamic workflows.
• scripts/scaffold-probe.py — Boris strict-workflow-scaffolding detector. Classifies each numbered item as scaffold, criterion, or branch, and caps on the scaffold count alone. Used by the Boris Alignment Check (quality-rubric §"The scaffolding discriminator") and freshen §4b.
• scripts/probe-trigger.py — Trigger-mode measurement tool. Adapted from anthropics/skills skill-creator/scripts/run_eval.py. Spawns claude -p subprocesses against a synthetic slash-command and parses stream-json for Skill/Read tool_use events to compute per-query trigger rate. Supports stratified train/test split, configurable runs-per-query, threshold, and parallelism.