Post eval

Skill SamyakJhaveri/loam/seed/_research/skills/post-eval

Copier template that bootstraps AI-agent-optimized project setups: layered context routing, curated skills, and an enforced validation gate — synced across projects.

Install

npx -y skills add SamyakJhaveri/loam --skill post-eval

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Post-batch analysis pipeline that runs after an eval batch completes. Use after /eval-run or /overnight-eval finishes. Verifies results, runs analysis scripts, refreshes dashboard, writes summary. Does NOT launch new eval runs.

SKILL.md

2.8 KB, as published. Nobody here has run it

Post-Eval Pipeline

After an eval batch completes, run the full post-processing chain: verify results, analyze, classify, refresh dashboard, and generate a summary. Each stage validates the previous stage's output before proceeding.

Trigger: When user types /post-eval or /post-eval <model-name>

Arguments

$ARGUMENTS — optional model name filter. If omitted, analyze all models.

Critical Rules

Ground truth is result files, not summaries. Always verify claims against the actual result files on disk.
Exclusion list is non-negotiable. Apply known-failing exclusions consistently.
Never assume "all clean" without spot-checking. Check 5-10 results manually.
Classify failures granularly. Different failure types have different root causes and implications for the paper.
Check experiment configuration explicitly. Don't assume parameters are correct.

Workflow

Step 0: Pipeline Integrity Check

Before running analysis, verify the evaluation operated correctly:

Status consistency across result fields
Configuration verification (parameters match expectations)
Coverage check (expected tasks all present)
Exclusion list applied correctly

If any check fails, report before proceeding.

Step 1: Verify Results Exist

Count result files per model/configuration. If no results, stop.

Step 2: Run Analysis Scripts

Execute the project's analysis pipeline. Verify outputs were created.

Step 3: Classification (if applicable)

Run any classification or categorization scripts. Verify output.

Step 4: Refresh Dashboard (if applicable)

Update any visualization or dashboard files with new results.

Step 5: Generate Summary

Write a structured summary to the project's docs directory:

Overall pass rates by model
Configuration breakdown
Notable findings and anomalies
Gaps in coverage

Step 6: Report & Suggest Next Steps

=== POST-EVAL PIPELINE COMPLETE ===

Step 0: Integrity check  - PASS / FAIL
Step 1: Results verified  - <N> models, <N> total files
Step 2: Analysis          - PASS / FAIL
Step 3: Classification    - PASS / FAIL / SKIP
Step 4: Dashboard         - PASS / FAIL / NO_CHANGES
Step 5: Summary written   - <path>

Suggested next steps:
  [ ] Review summary for anomalies
  [ ] Run /validate before committing
  [ ] Update paper sections if relevant

Error Handling

If any step fails:

Report the specific error
Mark that step as FAIL
Continue to next step if possible (steps are mostly independent after Step 2)
Do NOT retry automatically — let the user decide