Skill eval

Skill PIXARTSeu/Synapse/packages/codegraph/data/skill/skill-eval

Self-improving AI brain for Claude Code & Desktop — 28 MCP tools, 253 skills, collective memory, project tracking, work logs. One server, all your sessions share the same knowledge. Deploy on Coolify in 2 minutes.

Install

npx -y skills add PIXARTSeu/Synapse --skill skill-eval

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
8 stars8 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Runs and analyzes eval test cases for Claude skills. Use when asked to "test a skill", "check if this skill works", "run evals", "verify skill quality", or after creating/modifying a skill.

SKILL.md

2.7 KB, as published. Nobody here has run it

Skill Eval Runner

Analizza e verifica la qualità delle skill tramite i file evals/.

Overview

Esegue il ciclo RED-GREEN-REFACTOR adattato alle skill:

RED: verifica che la skill risolva un problema reale
GREEN: conferma che trigger e output siano corretti
REFACTOR: suggerisce miglioramenti basati sui test

Process

Step 1: Locate Evals

ls .Claude/skill/{skill-name}/evals/

Cerca: trigger_evals.json, evals.json

Step 2: Validate Trigger Accuracy

Leggi trigger_evals.json e verifica:

Le query in should_trigger matchano la description della skill?
Le query in should_not_trigger sono sufficientemente diverse?
Ci sono abbastanza test cases? (minimo 5 per should_trigger)

Report:

Trigger Analysis for {skill-name}:
✅ should_trigger: {n} cases — coverage seems adequate
⚠️  should_not_trigger: {n} cases — consider adding more negative cases

Step 3: Validate Quality Evals

Leggi evals.json e verifica:

I test cases coprono i casi d'uso principali?
Gli expected_output_contains sono realistici?
Ci sono test per edge cases?

Step 4: Run Script Validation

Se la skill ha scripts/:

bash .Claude/scripts/run_evals.sh 2>&1

Step 5: Generate Report

SKILL EVAL REPORT: {skill-name}
================================
Trigger evals: {n} should_trigger, {m} should_not_trigger
Quality evals: {n} test cases
Scripts: {present/absent}

Issues found:
- [lista problemi]

Recommendations:
- [lista suggerimenti]

Overall quality: {PASS/NEEDS_IMPROVEMENT/FAIL}

Examples

Example 1: Testare skill brainstorming

/skill-eval brainstorming
→ Legge .Claude/skill/brainstorming/evals/
→ Analizza trigger_evals.json: 7 should_trigger, 4 should_not_trigger
→ Analizza evals.json: 3 test cases
→ Report: PASS — coverage adeguata

Example 2: Nuova skill appena creata

/skill-eval my-new-skill
→ Legge .Claude/skill/my-new-skill/evals/
→ Trova solo 2 trigger cases
→ Report: NEEDS_IMPROVEMENT — aggiungere almeno 3 trigger cases
→ Suggerisce: "Add these trigger phrases: ..."

Troubleshooting

evals/ directory non trovata

Causa: La skill non ha ancora evals Soluzione: Usa skill-creator per aggiungere evals, oppure copia dal template in skill-template-2.0/evals/

JSON syntax error in evals file

Causa: Trailing comma o caratteri speciali Soluzione: Valida con python3 -m json.tool {file}