agentsclimarketplace

Eval best practices

Skill justinramos101/agent-skill-kit/skills/eval-best-practices

Battle-tested Agent Skills for coding agents — source-grounded, failure-driven heuristics that audit and design real surfaces. Install with npx skills.

Install
npx -y skills add justinramos101/agent-skill-kit --skill eval-best-practices

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Agent evals — design or audit the measurement program for an AI agent or skill: evaluation contracts, production-shaped datasets, task fidelity, calibrated LLM judges, outcome and trajectory tests, repeated-trial reliability, held-out release gates, production-feedback handoffs, and activation/trigger evals. Use for 'design our agent evals', 'is our judge trustworthy', 'are these tasks valid', 'which reliability metric fits', 'gate a model swap', or 'design the incident-to-regression loop'. Do NOT use for designing agent SDK/tool surfaces, writing agent-readable docs, scaffolding repo CI/hooks, reviewing human unit/integration tests, or operating production monitoring, triage, or drift response — this skill designs the eval contract and handoff.

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

3.8 KB, as published. Nobody here has run it

Eval Best Practices

Design or audit the measurement program an AI agent or skill is judged by. Run to completion, stating assumptions; ask only on a genuine blocker such as no obtainable ground truth or no way to observe the intended outcome.

Core principle: A trusted-but-wrong instrument is worse than none. Start from the product decision and externally verifiable user outcome, then decompose failure modes and spend judge budget only on the subjective residual.

Invariants

  • Task Fidelity: an impossible, ambiguous, leaky, or state-contaminated task measures the harness, not the agent. Validate the task, verifier, information boundary, and clean environment before diagnosing the system.
  • Outcome Truth: verify the achieved external state first. Grade the path only where authorization, safety, required hand-offs, cost, latency, or retry behavior can hide behind success; never require one exact path unless that path is itself an invariant.
  • Vague Judge: define the positive class and measure the confusion matrix against human-labeled failures and passes. Calibration belongs to (prompt × model × rubric); changing any one invalidates it.
  • Stochastic Evidence / march of nines: per-step or single-run success is not end-to-end reliability. Repeat isolated run-level trials using the deployed retry model; report counts, uncertainty, slices, and baseline deltas. Treat p^N as a planning warning, not observed reliability.
  • God Gate: one aggregate hides which risk broke. Classify each metric as blocker, thresholded guardrail, north star, or diagnostic, with an owner and failure path.

Workflow

  1. Define the evaluation contract: the decision, user job, system boundary, observable success state, unacceptable failures, release roles, and owners.
  2. Discover actual failure modes: inspect realistic traces with domain-aware reviewers before automating the taxonomy; then count frequency, severity, and affected slices.
  3. Validate the instrument: separate representative, challenge, and adversarial data; reject skeleton fixtures; isolate trials; keep user intent and expected outcome constant while the system under test changes.
  4. Load the matching branch: references/suite-design.md for task/data/scorer/reliability design, references/judges.md, references/trajectories.md, references/activation-evals.md, or references/program.md for the production-feedback contract.
  5. Build the smallest layered stack that can gate the decision: deterministic outcome/invariant checks, focused calibrated judges for semantic residuals, and targeted human review. Preserve per-check evidence before aggregation.
  6. Emit versioned artifacts: tasks, environments, scorers, trial policy, thresholds, slice tags, owners, rollback/escalation rules, and the coverage deliberately not added. In an audit, lead with whatever lets a material regression ship silently.

Self-review loops cap at 3 rounds; when a judge and a deterministic check disagree, stop and flag it — never tune to the judge.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.