Eval audit

Skill Goodeye-Labs/truesight-mcp-skills/skills/eval-audit

Agent skills for the Truesight MCP. Step-by-step workflow playbooks for scoring inputs, building live evaluations, error analysis, and the review loop. Works with Claude Code, Cursor, ChatGPT, VS Code, Windsurf, and any client that supports the agent skills standard.

Install

npx -y skills add Goodeye-Labs/truesight-mcp-skills --skill eval-audit

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

7 stars7 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Audit an existing evaluation workflow and produce severity-ranked findings with concrete next actions. Use when inheriting an eval setup, diagnosing quality regressions, or checking LLM evaluation process maturity.

SKILL.md

2.8 KB, as published. Nobody here has run it

Eval Audit

Audit LLM evaluation practice and route gaps to the right skills.

Interactive Q&A protocol (mandatory)

<HARD-GATE> BEFORE the first scoping question, search for a structured question tool (e.g., `AskUserQuestion` or similar interactive widget) and load it. Use that tool for EVERY scoping question. Fall back to plain-text lettered options ONLY if no such tool exists in the environment. </HARD-GATE>

Ask one question at a time using the structured question tool (loaded per the HARD-GATE above).

Example question structure:

What should this audit prioritize first?
A) Live evaluation quality and coverage
B) Error analysis maturity
C) Review and promotion loop health
D) End-to-end process health

Rules:

One question per message.
Use the structured question tool for every question. Structure each with a short header, 2-4 options with labels and descriptions, and place the recommended option first. Do not add "(Recommended)" or similar annotations to option labels.
Ask one follow-up only if ambiguity remains.

Inputs and evidence

Collect available evidence from Truesight first:

datasets and dataset rows
live evaluations
evaluation runs/results
review queue items
existing evaluation criteria and deployment patterns

If evidence is missing, record that as a finding.

Diagnostic areas

Evaluation coverage and quality dimensions
Error analysis practice and category quality
Review and promotion workflow discipline
Template usage versus custom needs
Operational hygiene (verification, reruns, iteration cadence)

Report format (mandatory)

For each finding, include:

### <Finding title>
Status: Problem exists | OK | Cannot determine
Evidence: <specific evidence from Truesight context>
Severity: critical | high | medium | low
Recommended skill: <one of current skill set>
Next command: <concrete instruction to run next>

Order findings by severity and impact.

Severity rubric

critical: likely causes incorrect go/no-go decisions or severe user harm
high: frequent quality failures or missing control loops
medium: meaningful process weakness with moderate impact
low: optimization opportunity, documentation, or ergonomics issue

Handoff map

Missing or weak failure taxonomy -> error-analysis
Missing live evaluation coverage -> create-evaluation or bootstrap-template-evaluation
Review backlog or low judgment throughput -> review-and-promote-traces
Unclear starting path -> truesight-workflows

Guardrails

Keep scope within current Truesight MCP capabilities.