Product ai evals
Skill jpoindexter/product-management-skills/skills/product-ai-evals
Operational product-management skills and a /pm router for Codex and Claude.
npx -y skills add jpoindexter/product-management-skills --skill product-ai-evalsAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- 14 days oldThe repository was created 14 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Design an AI-product evaluation system connecting task quality, user outcomes, failure taxonomies, datasets, human judgment, automated checks, release thresholds, and production monitoring. Use when defining AI quality, comparing models or prompts, deciding launch readiness, or diagnosing regressions.
SKILL.md
2.8 KB, as published. Nobody here has run it
Product AI Evals
Evaluate the end-to-end product task, not only a model metric. Connect offline tests, human judgment, controlled exposure, and production outcomes.
Inputs
- User task, workflow, and intended outcome
- Output types and acceptable variation
- Failure consequences and affected segments
- Current system, baseline, and alternatives
- Representative data and privacy constraints
- Cost, latency, and review budgets
Workflow
- Decompose the workflow into observable task steps and outcomes.
- Create a failure taxonomy covering incorrectness, omission, unsupported content, refusal, bias, unsafe action, poor recovery, latency, cost, and user misunderstanding.
- Build a representative evaluation set from real task distributions, difficult cases, high-impact cases, and adversarial cases. Track provenance and consent.
- Define rubrics with examples of pass, partial, and fail; use pairwise comparison when absolute scoring is unreliable.
- Combine deterministic checks, model or heuristic graders, expert review, and user outcome measures according to consequence.
- Establish baseline, candidate, segment slices, confidence intervals where appropriate, and regression budget.
- Set release thresholds and non-compensable safety or policy gates.
- Validate grader agreement and inspect disagreements; do not treat an automated judge as ground truth.
- Connect offline results to staged production monitoring and feedback.
Output contract
Return task definition, failure taxonomy, dataset plan, rubric, evaluator mix, metric definitions, segment slices, baselines, thresholds, release rule, monitoring links, and unresolved limitations.
Quality gate
- Evaluation examples match real usage and consequential edge cases.
- Product outcomes and model behavior are both measured.
- Aggregate scores cannot hide critical failure categories.
- Grader validity and human disagreement are monitored.
- Cost and latency are included in quality.
Avoid
- One generic “accuracy” score
- Testing only happy-path prompts
- Reusing production feedback without provenance review
- Optimizing to a judge without spot checks
- Launching because the average improved while a critical segment regressed
Source grounding
Operational synthesis informed by success-metric, trade-off, validation, and launch practices in Building AI-Powered Products and evaluation and production-system principles in Designing Machine Learning Systems.