Evaluate plugin or skill

Evaluate Plugin or Skill

A plugin is ready to share when its structure is correct, its activation is reliable, its content is disciplined, its documentation answers the three install-time questions, and its anti-patterns are absent. This workflow checks all five dimensions against the same standards skill-foundry and plugin-dev enforce — and returns a single verdict instead of a wall of unsorted feedback.

When to use this workflow

• You finished a plugin or skill and want to know if it is shippable
• You are reviewing a third-party plugin before installing or recommending it
• You drafted a SKILL.md and want to know whether the description will actually trigger
• You want a structured second opinion before publishing to a marketplace
• You need feedback that is ranked by priority — not a flat list of nitpicks

When NOT to use this workflow

• Building a new plugin from scratch → use build-a-plugin
• Extending or fixing an existing plugin → use update-a-plugin
• Authoring a single new SKILL.md → use write-your-own-skill
• Measuring runtime agent quality (eval suites for production agents) → use evaluate-and-improve-agent
• Fixing a single broken validation check → run plugin-validation directly

Prerequisites

Install the underlying plugins so the audit has real depth:

/plugin install plugin-validation@skillstack
/plugin install plugin-evaluation@skillstack
/plugin install skill-foundry@skillstack
/plugin install plugin-documenter@skillstack

Without these installed, the workflow still runs as a checklist — but the depth and the live eval harness disappear.

Core principle

A verdict beats a list. Ten unranked findings overwhelm; one verdict with three ranked actions ships. The audit sorts every finding by severity (blocker, gap, polish) and folds them into one of three outcomes: SHIP, IMPROVE, or REWORK. Authors get a clear next step instead of a homework pile.

Secondary principle: measure activation, do not assume it. Most plugins fail in the wild because their description does not trigger reliably — not because their content is bad. Trigger evals are non-negotiable; if none exist, this workflow generates seed evals and runs them before issuing a verdict.

Third principle: eat the dog food. This skill audits other skills against skill-foundry's anti-patterns. It must itself avoid those anti-patterns — lean SKILL.md, references on demand, decision tables over prose.

The phases

Phase 1 — Intake and classification

Detect what the user provided. Inputs come in three forms: a GitHub URL, a local plugin directory, or a single SKILL.md path.

Classify the artifact:

• Plugin — has .claude-plugin/plugin.json and a skills/ (or hooks/, mcp-servers/, agents/, commands/) directory
• Single skill — has a single SKILL.md with frontmatter, no plugin manifest
• Ambiguous — neither — ask the user before proceeding

Inventory components without deep-reading: count skills, hooks, MCP servers, subagents, commands. Note size of each SKILL.md in lines. This inventory drives which phases are relevant.

Output: a one-paragraph summary of what is being evaluated and which phases will run.

Phase 2 — Structural validation (plugin-validation)

Load the plugin-validation skill.

Check, for plugins:

• plugin.json has name, version, description, author, license
• Each skills/<name>/SKILL.md exists and has YAML frontmatter
• name in frontmatter equals the directory name
• All referenced files (references/*.md, scripts/*.py, etc.) exist on disk
• Plugin name and version follow conventions of the target marketplace
• No SKILL.md exceeds 500 lines (skill-foundry hard limit)

For single skills, run a subset: frontmatter validity, name field present, referenced files exist.

If plugin-dev/scripts/validate_plugins.py is available, run it for objective output. Otherwise, run the checklist by hand.

Severity classification:

• Blocker — manifest missing, name mismatch, frontmatter invalid, broken reference
• Gap — missing optional but expected fields (keywords, repository URL)
• Polish — convention drift (kebab-case violations, unusual naming)

Output: structural score (blocker count, gap count, polish count).

Phase 3 — Skill content audit (skill-foundry)

Load the skill-foundry skill.

For each SKILL.md, audit against the five canonical anti-patterns:

Anti-pattern	Symptom	Fix
Reference Illusion	SKILL.md cites references that do not exist or are never loaded	Inline the content or remove the citation
Description Soup	Frontmatter description is generic prose without trigger keywords or NOT clauses	Rewrite with concrete trigger phrases + explicit NOT clause
Template Theater	Sections present because the template said so, not because the skill needs them	Delete unused sections
Everything Skill	Tries to handle multiple unrelated capabilities	Split into multiple skills, one per capability
Orphaned Sections	Sections never referenced from anywhere — readers cannot find them	Either link to them or delete them

Also check the structural quality bar:

• Description has trigger phrases AND a NOT clause (target: ≥3 trigger keywords, ≥2 NOT cases)
• SKILL.md ≤500 lines (≤200 preferred for activation skills)
• Decision tables, not paragraphs, for routing logic
• Progressive disclosure: heavy content lives in references/, not inline

Severity classification:

• Blocker — Everything Skill, Reference Illusion, description with no trigger keywords
• Gap — Description Soup, no NOT clause, missing decision tables
• Polish — Template Theater, mild Orphaned Sections

Output: per-skill anti-pattern findings with severity.

Phase 4 — Activation evaluation (plugin-evaluation)

Load the plugin-evaluation skill.

If skills/<skill>/evals/trigger-evals.json exists, run the harness:

python3 plugin-dev/scripts/run_eval.py \
  --plugin-dir <path> --skill <name> --mode trigger

If no trigger evals exist, this is itself a finding — but do not stop. Generate seed evals: 5 positive cases (varied phrasings of when the skill should activate) and 5 near-miss negatives (queries that look related but should NOT trigger). Run those.

Targets:

• Positive trigger rate ≥ 90%
• Negative no-trigger rate ≥ 95%

Severity classification:

• Blocker — positive trigger rate < 60%, or negative no-trigger rate < 80% (skill collides with other skills)
• Gap — positive 60–89%, or negative 80–94%
• Polish — positive 90–94%, or negative 95–98%
• No finding — positive ≥ 95% AND negative ≥ 99%

Output: trigger-rate table per skill.

Phase 5 — Output evaluation (plugin-evaluation, optional)

If skills/<skill>/evals/evals.json exists, run output evals to check whether the skill produces correct results when activated. Otherwise, mark as a gap and proceed — output evals are nice-to-have for shipping, mandatory for production-critical plugins.

Severity classification:

• Blocker — output pass rate < 70% on existing evals
• Gap — no output evals exist for a plugin marketed as production-ready
• Polish — output pass rate 70–89%

Output: output-eval pass rate (or absence note).

Phase 6 — Documentation and shareability

The best plugin documentation answers three questions in install order: what problem does this solve, how do I install it, what can I do with it. Check whether the README answers these in that order.

Checklist:

• README opens with a problem statement, not "this plugin provides…"
• Installation block exists and works as written
• At least three concrete example prompts, scenarios, or invocations
• License and attribution footer present
• CHANGELOG.md exists with at least one version entry
• Plugin manifest keywords are searchable (matches actual use cases, not generic words like "tools" or "ai")

Severity classification:

• Blocker — no README, broken install instructions, no license
• Gap — README leads with capabilities instead of the problem; no example prompts; no CHANGELOG
• Polish — keyword stuffing, missing screenshots for visual plugins

Output: documentation gap list.

Phase 7 — Verdict synthesis

Score each dimension 0–3:

Score	Meaning
3	Zero blockers, zero gaps
2	Zero blockers, ≤2 gaps
1	Zero blockers, 3+ gaps OR 1 blocker
0	2+ blockers OR critical activation failure

Aggregate the five scores (structure, content, activation, output, documentation) into one verdict:

• SHIP — every dimension ≥ 2 AND total ≥ 12. Worth sharing as-is or with cosmetic polish.
• IMPROVE — one or two dimensions at 1, none at 0. Concrete, addressable fixes. List the top three ranked actions; the author can ship after addressing them.
• REWORK — any dimension at 0, OR three+ dimensions at 1, OR positive trigger rate < 60%. The plugin needs a structural revisit; suggest restarting from build-a-plugin (for plugins) or write-your-own-skill (for single skills).

Output: a one-screen verdict report with:

1. Verdict (SHIP / IMPROVE / REWORK) — single word, top of the report
2. Dimension scores table (5 rows, score + one-sentence reason each)
3. Top three ranked actions (only the ones that move the verdict — skip polish-only items unless the verdict is already SHIP)
4. If REWORK: which workflow to restart from and why

Gates and failure modes

Gate	Failure mode	Recovery
Phase 1 classification ambiguous	Cannot tell what is being audited	Ask the user before running phases 2–7
Phase 2 blocker found	Plugin will not load	Stop — return a REWORK verdict with structural fixes only
Phase 4 trigger evals missing AND skill description is the only signal	Cannot measure activation	Generate seed evals before issuing verdict; do not skip Phase 4
Phase 7 produces SHIP without trigger eval data	Verdict is unfounded	Downgrade to IMPROVE pending eval evidence
Author asks "just tell me if it is good"	Tempting to skip phases	Run all phases — verdict is only credible if every dimension was checked

The most common failure mode is issuing SHIP without measuring activation. If you cannot point to a trigger pass rate, you have not evaluated the plugin — you have read it.

Output artifacts

• A verdict report (SHIP / IMPROVE / REWORK) with dimension scores and ranked actions
• Optional: generated trigger-evals.json seed file if none existed
• Optional: a one-page diff between the plugin's current state and shippable state, suitable for handing back to the author

Related workflows and skills

• build-a-plugin — when REWORK is the verdict for a plugin, restart here
• write-your-own-skill — when REWORK is the verdict for a single skill
• update-a-plugin — when IMPROVE is the verdict and the author wants a guided fix loop
• plugin-validation, plugin-evaluation, skill-foundry, plugin-documenter — the underlying skills this workflow composes

---

Workflow part of skillstack-workflows by Viktor Bezdek — licensed under MIT.