Overnight eval

Skill SamyakJhaveri/loam/seed/_research/skills/overnight-eval

Copier template that bootstraps AI-agent-optimized project setups: layered context routing, curated skills, and an enforced validation gate — synced across projects.

Install

npx -y skills add SamyakJhaveri/loam --skill overnight-eval

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Long-running evaluation campaign with tmux isolation. Use when running any eval batch exceeding ~30 minutes, canonical benchmark runs, or when the user will be away during execution. Pre-flight verification, tmux launch, monitoring, post-flight analysis. NOT for: short interactive runs (use eval-run instead).

SKILL.md

3.2 KB, as published. Nobody here has run it

Overnight Eval Campaign

Use when launching a long-running evaluation batch that needs tmux deployment or should run unattended (overnight, weekend).

Trigger: When user types /overnight-eval with optional arguments.

Arguments

$ARGUMENTS — shorthand or explicit flags for the eval script. If omitted, prompt interactively.
--dry-run — run pre-flight only, do not launch
--session-name <name> — tmux session name (default: eval-campaign)
--alert-threshold <N> — error rate % that triggers alert (default: 50)

Iron Law

NO EVAL CAMPAIGN WITHOUT PRE-FLIGHT VERIFICATION.
Every API key, environment dependency, dataset, and eligibility check MUST pass
before the batch command is issued. No exceptions.

Anti-Rationalization Table

Excuse	Reality
"I'll check the API keys later"	A failed key means wasted hours and zero results
"The environment is probably fine"	2 seconds to verify. Hours to re-run
"I can run it in the foreground"	Terminal disconnect = killed process. Always tmux
"Resume will catch anything I miss"	Resume perpetuates pre-flight errors
"I'll analyze tomorrow"	Tomorrow you won't remember the parameters. Log now

Workflow

Phase 1: Pre-Flight Verification

Run ALL checks. Every check must pass before proceeding.

Environment: Verify venv/dependencies, Python version, key packages
Hardware (if applicable): GPU availability, memory, utilization
API Keys: Check only for selected models
Dataset/Source: Verify benchmark data exists and is accessible
Eligibility: Verify task list, apply exclusions

Display pre-flight summary and wait for user confirmation.

If --dry-run, stop after displaying summary.

Phase 2: Launch in tmux

mkdir -p <project-root>/logs/eval-staging
tmux new-session -d -s <session-name> -c <project-root>
tmux send-keys -t <session-name> "<eval-command> 2>&1 | tee logs/eval-staging/campaign_$(date +%Y%m%d_%H%M%S).log" Enter

Verify tmux session is running before proceeding. Do not write campaign logs directly under results/; keep raw run logs in staging until post-flight review.

Phase 3: Monitor (Periodic)

The user can check progress at any time:

Is the tmux session still alive?
How many results completed?
Current pass/fail rates
Error spike detection (alert if error rate exceeds threshold)

Phase 4: Post-Flight Analysis

After completion:

Run analysis scripts
Review staged campaign log for anomalies
Generate summary with results table
Alert user (desktop notification if available)

Phase 5: Handoff

Present results summary and recommended next steps:

Promote reviewed artifacts into results/ only through the approved results ingestion path
Refresh dashboard if applicable
Commit results
Update paper sections if relevant