Skill description tuning

A skill only helps if it activates. The description is the only text an agent sees at startup, so it carries the entire burden of triggering.

Scope this work honestly before starting. Rewriting a description takes minutes; running a proper eval takes an hour or more and needs a CLI agent. Ask which the user wants — do not silently do the cheap version and present it as the measured one.

1. Diagnose

The symptom points at the fix:

Symptom	Likely cause	Direction
Never triggers	Description says what, never when	Add explicit trigger contexts
Triggers only when named explicitly	No overlap with how users actually phrase the task	Add the user's vocabulary, not the domain's
Triggers on unrelated tasks	Too broad, or shares keywords with adjacent needs	Add boundaries — what it does not cover
Triggers inconsistently	Sits near the decision boundary	Measure before changing anything

One caveat that saves wasted effort: agents mostly consult skills for tasks needing capability beyond what they can already do. A one-step request like "read this PDF" may not trigger a PDF skill however good the description, because the agent can just do it. If the skill covers something the agent handles unaided, the problem is the skill's existence, not its description — say so, and hand off to skill-quality-audit.

2. Rewrite

Four principles, from agentskills.io:

• Imperative, not descriptive. "Use this skill when…", not "This skill does…". The agent is deciding whether to act.
• User intent, not implementation. The agent matches against what the user asked for, not the skill's internals.
• Be pushy. List the contexts explicitly, including ones where the user will not name the domain: "…even if they don't explicitly mention 'CSV' or 'analysis'."
• Stay concise. A few sentences to a short paragraph. The hard limit is 1024 characters, but every description is loaded in every session.

Before and after:

# Before
description: Process CSV files.

# After
description: >
  Analyze CSV and tabular data files — compute summary statistics,
  add derived columns, generate charts, and clean messy data. Use this
  skill when the user has a CSV, TSV, or Excel file and wants to
  explore, transform, or visualize the data, even if they don't
  explicitly mention "CSV" or "analysis."

The improvement is in both directions at once: more specific about what (summary stats, derived columns, charts, cleaning), broader about when (CSV, TSV, Excel; without explicit keywords).

Run skill-validation after any rewrite. It catches the 1024-character limit and the phrasing failures (AS204 missing trigger language, AS205 self- description, AS206 startup-context cost).

3. Build the eval set

Only when the user wants this measured.

Aim for ~20 queries: 8–10 should-trigger, 8–10 should-not-trigger. Start from assets/eval_queries.template.json.

The two failure modes in eval design — writing positives so obvious that any description passes, and negatives so unrelated that they test nothing — are what make an eval useless. references/eval-design.md covers how to avoid both, with worked examples of strong and weak queries. Read it before writing the set.

The short version: the useful positives are ones where the skill would help but the connection is not obvious from the query. The useful negatives are near-misses that share keywords with the skill but need something else.

4. Measure

bash scripts/trigger_eval.sh train_queries.json my-skill-name

The script runs each query N times and reports a trigger rate per query, because model behavior is nondeterministic — a single run tells you nothing. Default 3 runs; a query passes at a trigger rate above 0.5 for positives, below it for negatives.

Read scripts/trigger_eval.sh --help first. It ships configured for Claude Code's JSON output and has one function to replace for any other client.

Split the set 60/40 into train and validation before the first run. Keep both proportionally mixed, and keep the split fixed across iterations. Use only train failures to guide changes; validation tells you whether the changes generalize rather than overfit to your own phrasings.

5. Iterate

1. Evaluate on both sets. Train guides; validation judges.
2. Find the train failures. Which positives missed? Which negatives fired?
3. Rewrite:
   • Positives failing → too narrow. Broaden the contexts.
   • Negatives firing → too broad. Add boundaries, or say what it does not do.
   • Do not paste keywords from failed queries. That is overfitting. Find the general category those queries represent and address that.
   • Stuck after several rounds? Try a structurally different description rather than more incremental tweaks.
4. Repeat. Five iterations is usually enough.
5. Select by validation pass rate, not by recency. The best version is often not the last one — later iterations frequently overfit. Keep every version and its scores so this comparison is possible.

If nothing improves across five iterations, suspect the queries — too easy, too hard, or mislabelled — rather than the description.

6. Confirm

Write 5–10 fresh queries that were never part of the optimization and run those. Never-seen queries are the only honest check that the description generalizes.

Report the trigger rates, both sets, and which iteration won. A description change without numbers is a guess — say so when that is what it is.

Gotchas

• A trigger eval is expensive. 20 queries × 3 runs = 60 agent invocations per iteration, and five iterations is 300. Tell the user before starting.
• Stop runs early where the client allows it. Once the agent has either consulted the skill or started working without it, the outcome is known.
• The name matters too. A vague name works against a good description. Renaming has costs — it changes the required directory name (AS106) and breaks anything referencing the skill — so raise it, do not do it.
• Other installed skills affect triggering. A near-duplicate description on another skill produces inconsistent rates that no rewrite fixes. Check what else is installed before blaming the description.
• Do not tune a skill that should not exist. If the agent handles the task well unaided, better triggering makes things worse, not better.