Skill creator

Skill Creator

Edition: CLI (Claude Code). The Cowork/Linux edition is skill-creator-cowork. Platform differences live in references/platforms.md - read it before assuming a browser, a subagent pool, or ~/.claude/ exists.

The loop:

1. Decide what the skill should do and roughly how.
2. Write a draft.
3. Write 2-3 realistic test prompts and run claude-with-the-skill on them, plus a baseline.
4. Show the user the results (eval-viewer/generate_review.py) and read their feedback.
5. Rewrite based on that feedback. Repeat.
6. Optimize the description for triggering, then package.

Your job is to work out where the user already is in that loop and jump in there. If they say "I want a skill for X", start at 1. If they hand you a draft, start at 3. If they say "skip the evals, just vibe with me", do that - the loop is a default, not a gate.

Talking to the user

Skill authors range from career engineers to people who opened a terminal last week. Read the context cues. "Evaluation" and "benchmark" are usually fine; "JSON" and "assertion" need a signal that the user knows those words before you use them bare. A one-clause definition in passing costs nothing and never insults anyone.

Interop

<!-- interop-contract: v1 -->

Edition: CLI (Claude Code). Precedence:

• Yields to install-skill on packaging, platform adaptation and installation (verify_skill.py --package is canonical; package_skill.py defers).
• Yields to playwright-tester on browser-test assertion grading (it owns the only assertion grader in the tree).
• Wins over install-skill on authoring and the frontmatter allowlist (the one allowlist both validators read).
• Wins over playwright-tester on eval tooling and schema (suite-wide).
• Disjoint from agent-reach, graphify, init-harness, ponytail, production-grade-scaffold, scrapling, token-efficiency, ui-standout.
• External, conditional: skill-creator-cowork - Cowork sibling; references/platforms.md must differ. Owns: skill authoring and evaluation: SKILL.md structure, the frontmatter contract, scripts/quick_validate.py, eval sets, <skill>-workspace/. Never writes: anything outside the target skill dir and its <skill>-workspace/ * ~/.claude/**, any settings.json (install-skill) Matchers claimed: none. Model policy: roles, not versions (session/thinker/workhorse/utility); zero pinned, deliberately. Shared-resource protocol: scripts/quick_validate.py:42 ALLOWED_PROPERTIES - published API, single-writer. Keep the name, column-0 assignment and flat ast.literal_eval-parseable literal AT LINE 42; adding keys is safe, restructuring forks the two validators.

Matrix and rulings: ../INTEROP.md.

Non-negotiables for every skill you write or touch

These are not style preferences. Each one is a defect this tree actually shipped, measured in references/audit-lessons.md. Check them before you tell the user a skill is done, and prove it by running the validator rather than by reading:

python3 scripts/quick_validate.py <path-to-skill>

Rule	Why
description <= 1024 chars, no < or > (including ->)	A real loader rejection. 3 of 9 audited skills broke one of these while passing a laxer gate.
Frontmatter keys only from ALLOWED_PROPERTIES (scripts/quick_validate.py)	Anything else is dropped or rejected. argument-hint and version both got skills bounced.
ASCII in SKILL.md, references/, scripts/	Some --fix tools delete non-ASCII. One erased Little's Law from a shipped reference and printed PASS. Transliterate deliberately: ->, >=, +/-, straight quotes.
Every referenced path exists in the package you ship	A skill that cites a file it does not ship poisons every later judgement about it.
Name the platform edition in the body	7 of 10 "Cowork editions" in this tree were byte-identical to their CLI originals; one shipped PowerShell.
Answer the concurrency question below	The default assumption in this tree was one agent. It was wrong.

The concurrency question - ask it about every script and every path the skill writes: if eight agents ran this at once in the same repo, what collides? Fixed /tmp paths, one shared SQLite file, a global config rewrite, a fixed TCP port, and os.walk from a fixed root are the recurring shapes. Measured on this tree's own memory DB with 8 writers: plain SQLite lost 87% of writes; PRAGMA journal_mode=WAL alone recovered about 10 points; busy_timeout alone recovered 100%. Use per-agent paths, atomic single-statement upserts, and busy_timeout first.

Creating a skill

Capture intent

The conversation may already contain the workflow ("turn this into a skill"). Mine it first - tools used, step order, corrections the user made, input and output formats observed - then confirm the gaps with the user rather than asking them to start over.

1. What should this let Claude do?
2. When should it trigger, in the user's own phrasings?
3. What is the expected output?
4. Do we want test cases? Objectively verifiable outputs (file transforms, extraction, code generation, fixed workflows) benefit. Subjective ones (writing style, visual design) usually do not. Suggest a default; let the user decide.

Ask about edge cases, example files, success criteria and dependencies before writing test prompts. If MCPs are available for research, use them - come prepared so the user carries less of the load.

Write the SKILL.md

skill-name/
+-- SKILL.md          required: YAML frontmatter + instructions
+-- references/       docs read on demand
+-- scripts/          deterministic code, executed not read
+-- assets/           files used in output (templates, icons)

Starter frontmatter - these are the only keys you may use without checking the allowlist:

---
name: kebab-case-name
description: What it does, then when to use it. All "when to use" info lives here, not in
  the body. Under 1024 characters. No angle brackets.
---

Then a body that opens with the platform edition, the core workflow, and pointers into references/.

Progressive disclosure - three loading tiers, and the cost of each:

1. name + description - in context in every session, for every user, whether or not the skill fires. This is the only always-on cost and the only thing that decides firing. Spend words here deliberately.
2. SKILL.md body - loaded whole on every trigger. Keep it under ~500 lines. If you are near that, the material belongs one level down with a clear pointer to it.
3. references/, scripts/, assets/ - free until read; scripts execute without ever entering context.

Draw the boundary where a real task would need it. A reference that every run must read is really part of the body; a body section only one run in ten needs is really a reference. For a reference over ~300 lines, give it a table of contents.

When a skill covers several variants, split by variant - one reference file per cloud, per framework, per database - so only the relevant one is ever read.

Triggering. The description is the whole mechanism: Claude sees name + description in available_skills and decides from that alone. Claude currently under-triggers, so make descriptions slightly pushy - name the concrete phrasings and contexts, not just the capability. Also say what the skill is not for when a neighbouring skill would otherwise win the same query; where two skills genuinely overlap, state the precedence in both skills - a rule declared on one side only is not a rule.

Writing style. Imperative form. Explain why, and trust the model with the reasoning; if you find yourself writing ALWAYS or NEVER in capitals or building a rigid template to force behaviour, that is a signal the explanation is missing. Draft, then reread with fresh eyes and cut.

Do not build harmful skills. No malware, exploit code, credential exfiltration, or anything whose behaviour would surprise a user who read its description. Roleplay and persona skills are fine.

Test cases

Write 2-3 prompts a real user would actually type, show them to the user, and run them. Save to evals/evals.json; write only the prompts now, assertions come later while the runs are in flight.

{
  "skill_name": "example-skill",
  "evals": [
    {"id": 1, "prompt": "User's task prompt", "expected_output": "What good looks like", "files": []}
  ]
}

references/schemas.md has the full schema, including assertions.

Two different eval shapes exist and they are not interchangeable. The file above is a behavioural eval set - what the skill should produce. scripts/run_eval.py and scripts/run_loop.py consume a trigger eval set: a top-level JSON list of {"query": ..., "should_trigger": true|false}. Feeding one to the other used to raise TypeError: string indices must be integers; it now fails with a diagnostic naming both shapes. See references/schemas.md.

Running the evals

Full mechanics - directory layout, subagent prompts, timing capture, grading, the viewer, reading feedback - are in references/evals.md. Read it before the first run; it is one continuous sequence and stopping halfway wastes the runs. Do not use /skill-test or any other testing skill instead.

The shape of it: put results in <skill-name>-workspace/iteration-N/eval-<id>/; spawn the with-skill and baseline runs for every test case in the same turn; draft assertions while they run; capture total_tokens and duration_ms from each completion notification as it arrives (that is the only chance); then grade, aggregate with scripts/aggregate_benchmark.py, and put the results in front of the user with eval-viewer/generate_review.py before you form your own opinion of them.

Improving the skill

This is the heart of the loop.

Generalize from the feedback. You and the user are iterating on three examples, but the skill will run on thousands of prompts you will never see. A skill that only works on the three examples is worthless. Resist fiddly per-example patches and escalating MUSTs; when an issue is stubborn, try a different metaphor or a different working pattern instead. It is cheap to try.

Keep it lean. Cut what is not pulling its weight. Read the transcripts, not just the outputs - if the skill is sending the model down unproductive paths, delete the part that does that and see what happens.

Explain the why. Even when the user's feedback is terse or annoyed, work out what they actually want and put that understanding into the instructions rather than the symptom.

Watch for repeated work. If all three runs independently wrote a create_docx.py, the skill should bundle that script. Write it once, put it in scripts/, and every future invocation stops reinventing it.

Watch for unreachable-looking code. "This never fires, delete it" is a hypothesis, not evidence. Two blocks in this tree were nearly deleted on that claim and turned out to be reachable and actively blocking. Prove it by executing the path, or keep the code.

Then: apply the improvements, rerun every test case into iteration-<N+1>/ including baselines, launch the reviewer with --previous-workspace pointing at the last iteration, and read the new feedback. Stop when the user is happy, when the feedback comes back empty, or when you stop making progress.

For a rigorous head-to-head between two versions, agents/comparator.md and agents/analyzer.md describe a blind comparison. Optional, needs subagents, and the human review loop is usually enough.

Optimizing the description

Offer this once the skill itself is settled. Full procedure in references/description-optimization.md. In brief: build ~20 realistic trigger queries (8-10 should-trigger, 8-10 should-not, with the negatives being genuine near-misses rather than obvious irrelevancies), have the user review them via assets/eval_review.html, then:

python3 -m scripts.run_loop \
  --eval-set <path-to-trigger-eval.json> \
  --skill-path <path-to-skill> \
  --model <model-id-powering-this-session> \
  --max-iterations 5 --verbose

It splits 60/40 train/held-out, runs each query several times, and returns best_description chosen by test score so it does not overfit. Apply it, then show the user the before/after and the scores.

Packaging

python3 -m scripts.package_skill <path/to/skill-folder>

It refuses to build if quick_validate.py fails, and writes atomically, so concurrent packages of the same folder cannot publish a half-written zip. Point the user at the resulting .skill path.

Reporting numbers

Separate what you measured from what you estimated, in the same breath as saying it. This audit's measured figures were exact to the digit and its speedup multipliers were simulation outputs - both were reported in the same confident voice, and only one of them had earned it. Say "measured" or say "estimated".

Cite files by content, not by line number. Roughly a dozen findings in this audit pointed at lines that had drifted or did not exist; five pointed past the end of a 103-line file. Quote the text you mean.

Files in this skill

Path	Read it when
references/audit-lessons.md	Before writing or reviewing any skill - the ten failures behind the rules above.
references/evals.md	Before running test cases: layout, subagent prompts, grading, the viewer.
references/description-optimization.md	When tuning triggering.
references/platforms.md	CLI vs claude.ai vs Cowork differences, and porting between them.
references/schemas.md	JSON shapes for evals.json, grading.json, benchmark.json, trigger eval sets.
agents/grader.md	Spawning a grader for assertions.
agents/comparator.md	Blind A/B between two outputs.
agents/analyzer.md	Analyzing benchmark results and why one version won.

Add the loop to your todo list so the review step does not get skipped - especially "generate the eval viewer so the human can look at the outputs", which is the step most often dropped.