Eval grader

Skill SamyakJhaveri/loam/seed/_research/skills/eval-grader

Copier template that bootstraps AI-agent-optimized project setups: layered context routing, curated skills, and an enforced validation gate — synced across projects.

Install

npx -y skills add SamyakJhaveri/loam --skill eval-grader

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Evaluation result grading and classification workflow. Use when classifying eval results into pass/fail categories, diagnosing failure patterns in batch results, computing pass rates with proper exclusions, or preparing result tables for papers. Handles the grading pipeline: load results, apply exclusions, classify failure modes, compute statistics, generate summary tables.

SKILL.md

2.4 KB, as published. Nobody here has run it

Eval Result Grader

Structured workflow for grading and classifying evaluation results from batch runs. Produces defensible statistics for paper reporting.

Trigger: /eval-grader or when classifying/summarizing eval batch results.

When to use

After /post-eval completes and you need to classify results
Computing pass rates for paper tables
Diagnosing systematic failure patterns across models
Comparing two models' results on the same task set

Arguments

<result-path> — path to a result JSON or directory of results
--model <name> — filter to specific model
--config <name> — filter to specific configuration

Grading Pipeline

Step 1: Load Results

Read result files from the project's results directory. Each result should contain at minimum: task identifier, model, configuration, and outcome status.

Step 2: Apply Exclusions

Apply known-failing case exclusions. The exclusion list should be maintained in known-issues.md or equivalent. Key rules:

Exclude cases where the task itself is known to be broken
Document the exclusion formula and denominator clearly
Report excluded count alongside statistics

Step 3: Classify Failure Modes

For non-passing results, classify into fine-grained categories relevant to the project's domain. Build a failure taxonomy with:

Category and subcategory
Typical root cause
Count and percentage of total failures

Step 4: Compute Statistics

Report per model and configuration:

Pass rate:       PASS / N_valid x 100
Failure breakdown: <category>: X%, <category>: Y%

For papers: report both single-sample and multi-sample pass rates if applicable.

Step 5: Generate Summary Table

Output a markdown table suitable for paper inclusion.

Verification Gates

Before reporting any statistic:

Confirm exclusion list was applied
Confirm denominator matches expected task count
Cross-check at least 2-3 individual results manually
Flag anomalies (e.g., tasks that pass at baseline but fail with augmentation)