Eval run

Skill SamyakJhaveri/loam/seed/_research/skills/eval-run

Copier template that bootstraps AI-agent-optimized project setups: layered context routing, curated skills, and an enforced validation gate — synced across projects.

Install

npx -y skills add SamyakJhaveri/loam --skill eval-run

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Launch a model evaluation batch. Use when running interactive/foreground eval runs — parameter collection, pre-flight checks, execution, and post-run analysis. NOT for: long-running batches that need tmux isolation (use overnight-eval instead).

SKILL.md

2.4 KB, as published. Nobody here has run it

Eval Batch Launcher

Launch a model evaluation batch with automatic parameter collection, exclusion checks, API key verification, and post-batch analysis.

Trigger: When user types /eval-run with optional arguments.

Arguments

$ARGUMENTS — optional shorthand or explicit flags for the eval script. Omit entirely to be prompted interactively.

Workflow

Phase 1: Parse & Collect

Extract parameters from $ARGUMENTS. Prompt for missing required values.

Parameter	Notes
Suite/dataset	Which benchmark suite or dataset to evaluate
Configuration	Model config, direction, or task variant
Models	Which models to evaluate
Samples	Number of samples per task
Resume	Whether to resume from previous partial run

Phase 2: Pre-flight

Verify project environment is active (venv, dependencies)
Verify API keys are set for selected models
Verify dataset/benchmark files exist
Check for known-failing cases to exclude
Display pre-flight summary and wait for user confirmation

=== EVAL BATCH PRE-FLIGHT ===
Dataset:      <name>
Models:       <model1>, <model2>, ...
Config:       <config details>
Exclusions:   <N> known-failing cases excluded
Total tasks:  <N models> x <N tasks> = <total>
API keys:     all verified / <which missing>

Proceed? (yes / no / modify params)

Phase 3: Execute

Run the evaluation script with collected parameters. Monitor live output. If many consecutive failures appear, surface them to the user and confirm before continuing.

Phase 4: Analyze & Report

After batch completes:

Run analysis scripts to generate summary
Display results table (pass rates per model, failure breakdown)
Surface notable failures or unexpected patterns
Suggest next steps (dashboard refresh, commit results)

Integration

Pairs with /post-eval for full post-processing pipeline
Pairs with /overnight-eval for long-running campaigns
Pairs with /eval-grader for result classification