Plugin audit

Skill build-with-dhiraj/ai-workflow-framework-portability-kit/Plugins/vercel-marketplace-source/.claude/skills/plugin-audit

Audit vercel-plugin performance on real-world projects. Extracts tool calls from Claude Code conversation logs, tests hook matching against actual inputs, identifies pattern coverage gaps, and checks plugin cache staleness. Use when asked to audit, test, or investigate plugin skill injection on a real project.From its SKILL.md

Install

npx -y skills add build-with-dhiraj/ai-workflow-framework-portability-kit --skill plugin-audit

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

2.9 KB, 621 tokens by cl100k_base, as published. Nobody here has run it

Plugin Audit

Audit how well vercel-plugin skill injection performs on real-world Claude Code sessions.

Workflow

1. Locate conversation logs

Find JSONL conversation logs for a target project:

ls -lt ~/.claude/projects/-Users-*-<project-name>/*.jsonl

The path uses the project's absolute path with slashes replaced by hyphens and a leading hyphen.

2. Extract tool calls

Parse the JSONL log to extract all tool_use entries. Each line is a JSON object with message.content[] containing type: "tool_use" blocks. Extract name and input fields. Group by tool type (Bash, Read, Write, Edit).

3. Test hook matching

Use the exported pipeline functions directly — do NOT shell out to the hook script for each test. Import from the hooks directory:

import { loadSkills, matchSkills } from "./hooks/pretooluse-skill-inject.mjs";
import { createLogger } from "./hooks/logger.mjs";

Call loadSkills() once, then matchSkills(toolName, toolInput, compiledSkills) for each tool call. This is fast and gives exact match results.

4. Identify gaps

Compare matched skills against what SHOULD have matched based on the project's technology stack. Common gap categories:

Path pattern gaps: Files that should trigger a skill but don't (e.g., src/db/schema.ts not matching vercel-storage)
Bash pattern gaps: Commands that should trigger but don't (e.g., missing package manager variants)
Dedup masking: Skills that matched but were deduped before injection
Budget/cap drops: Skills matched but dropped by the 12KB budget or 3-skill ceiling

5. Check plugin cache staleness

Compare the installed plugin cache against the dev version:

# Cache location
~/.claude/plugins/cache/vercel-labs-vercel-plugin/vercel-plugin/<version>/

# Compare skill content
diff <(grep 'pattern' skills/<skill>/SKILL.md) <(grep 'pattern' ~/.claude/plugins/cache/.../skills/<skill>/SKILL.md)

Check ~/.claude/plugins/installed_plugins.json for version and git SHA.

Report Format

Produce a structured report with:

Session summary: Project, date, tool call count, model
Match matrix: Table of tool calls × matched skills (with match type)
Coverage gaps: Unmatched tool calls that should have matched, with suggested pattern additions
Dedup timeline: Order of skill injections and what got deduped
Cache status: Whether installed version matches dev, with specific diffs

References

What ships with it: 2 files

8.2 KB alongside SKILL.md, 1 of them executable

references/

log-format.md1.7 KB

scripts/

batch-match.mjsruns6.5 KB

Gives 0 of the 12 instructions most test skills give in 621 tokens

Counted across 1,201 of the 2,096 authors here whose files we hold, read 2026-09-06

Write a failing test before writing codein 43 of 1201, across 36 files
Run the full test suitein 36 of 1201, across 35 files
Test only one variable per experimentin 34 of 1201, across 17 files
Read product marketing context before asking questionsin 34 of 1201, across 14 files
Mock external dependenciesin 34 of 1201, across 30 files
Define primary, secondary, and guardrail metricsin 33 of 1201, across 16 files
Pre-determine sample size before startingin 31 of 1201, across 14 files
Test behavior rather than implementationin 31 of 1201, across 29 files
Formulate a hypothesis before designing a testin 30 of 1201, across 13 files
Document every test hypothesis, variant, and resultin 29 of 1201, across 11 files
Use descriptive test function namesin 25 of 1201, across 21 files
Commit to the methodology without stopping earlyin 24 of 1201, across 8 files

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.