Authoring skills with evals

Skill vemodalen-x/VEMO_SKILLS/skills/governance/authoring-skills-with-evals

Public reusable skill hub for agent workflows

Install

npx -y skills add vemodalen-x/VEMO_SKILLS --skill authoring-skills-with-evals

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Author, evaluate, and improve skills with an eval-driven loop instead of by intuition. Use when creating a new skill, revising an existing one, judging whether a skill's description actually triggers, or tuning that description for better trigger accuracy. Runs a with-skill-vs-baseline behavioral eval with variance, a trigger-rate eval, and a train/test-split description optimizer via the repo's skill-creator harness, modeled on Anthropic's official skill-creator. Triggers include create a skill, write a SKILL.md, run a skill eval, benchmark a skill, description not triggering, improve a skill description, skill quality, eval-driven authoring.

SKILL.md

5.5 KB, as published. Nobody here has run it

Authoring Skills with Evals

Create and improve a skill the way you would tune a model: draft it, measure it on realistic prompts, and iterate on evidence — not on a hunch that "the description reads fine". This skill is the VEMO_SKILLS adaptation of Anthropic's official skill-creator meta-skill; it keeps that loop but binds it to this home's single-source model and its release scorer.

Two layers, do not confuse them.

tools/vemo_skills_check.py (the release scorer, threshold 9.5) lints a skill's shape — frontmatter, naming, catalog parity, decoupling. It cannot tell you whether the skill works.

tools/skill_creator.py (this skill's harness) measures behavior — does the description trigger, and does the body produce a better result than no skill at all. A skill is only "done" when it passes both.

When to use

Creating a new skill from scratch, or splitting/merging existing ones.
A skill under- or over-triggers (fires when it shouldn't, or stays silent when it should).
You want to optimize a description: for trigger accuracy without overfitting to a few queries.
Auditing a skill's authoring provenance (the .skill-validated.json marker).

The loop (eval-driven authoring)

See references/eval-loop.md for the full rationale and prompt-design rules. In short:

Draft the SKILL.md. Put all the "when to use it" signal in description: — it is the only text the model sees when deciding to invoke the skill. Write it in the imperative and make it a little "pushy": the model tends to under-trigger skills.
Write an eval set — 6-12 realistic prompts as JSON, each {"query": "...", "should_trigger": true|false}. Include hard negatives (near-misses that must NOT trigger) and messy, real-sounding positives.
Lint the shape: python3 tools/skill_creator.py validate skills/<cat>/<name>. On pass it writes .skill-validated.json with tier: "lint" — an honest marker that says only the shape was checked.
Measure triggering: python3 tools/skill_creator.py trigger-eval skills/<cat>/<name> --eval-set evals.json. Reports a per-query trigger rate over N runs (triggering is nondeterministic — one sample is noise).
Improve the description: python3 tools/skill_creator.py describe-improve skills/<cat>/<name> --eval-set evals.json --model <model>. Splits the eval set train/test, optimizes on train, and picks the winner by held-out test score so it cannot memorize the eval set.
Re-run until the trigger eval is clean and a with-skill run beats the no-skill baseline.

Commands

command	what it does	needs `claude` CLI
`validate <dir> [--no-marker]`	lint frontmatter + naming; write `.skill-validated.json` (tier=lint)	no
`trigger-eval <dir> --eval-set f.json`	measure description trigger rate (tri-state per run)	yes
`describe-improve <dir> --eval-set f.json --model M`	train/test-split description optimization	yes
`selftest` (`vemo-skills author-selftest`)	hermetic check of the deterministic core	no

The two model-in-the-loop commands degrade gracefully: if the claude CLI is not on PATH they report status: skipped and exit 0. An infrastructure gap is reported as skipped, never scored as "the description failed to trigger" — a run that could not execute is an error, excluded from the rate.

Provenance marker (honest tiers)

.skill-validated.json carries a tier:

lint — frontmatter + naming only. Does not claim a behavioral/trigger eval ran.
trigger — lint plus a trigger eval that met the pass threshold.

It also records validator_sha (sha256 of the harness itself, so a changed validator is detectable) and the git_commit. Never hand-edit the marker; regenerate it with validate.

Boundaries

Does not adopt. Producing/validating a skill is not the same as putting it in a project's toolset (a user-consent decision). Registration/placement is publishing-skills; naming conformance is naming-skills; this skill owns the eval loop and complements both.
Render-nothing / send-nothing. It writes only the skill under authoring and its marker; it never publishes, announces, or edits the catalog.
Zero project identity in any skill body (this home's red line) — the harness only reads shape and triggering, so keep evals free of project-specific values too.

References

references/eval-loop.md — the full loop, the two eval types (trigger vs output-quality), variance / baseline discipline, the train/test split, and description-writing rules.
Anthropic Agent Skills — best practices: https://docs.claude.com/en/docs/agents-and-tools/agent-skills/best-practices
Anthropic engineering — equipping agents with Agent Skills (the official skill-creator's home).