Inspect evaluation harness starter
A framework for discovering, compiling, and validating reusable skills for scientific agents.
npx -y skills add ma-compbio-lab/SkillFoundry --skill inspect-evaluation-harness-starterAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
SKILL.md
1.3 KB, as published. Nobody here has run it
Inspect Evaluation Harness Starter
Use this skill to run a deterministic Inspect evaluation harness over toy scientific-agent cases and compare a candidate solver against a weaker baseline.
What This Skill Does
- defines a small local Inspect task set without external model credentials
- evaluates two deterministic solver variants on the same cases
- writes machine-readable accuracy summaries plus Inspect log files for both runs
When To Use It
- when you need a runnable
evaluation-harnesses-for-scientific-agentsstarter - when you want a local Inspect example before wiring in real agents or model-backed solvers
- when you need a stable comparison harness for repository tests
Run
./slurm/envs/agents/bin/python skills/scientific-agents-and-automation/inspect-evaluation-harness-starter/scripts/run_inspect_evaluation_harness.py \
--cases skills/scientific-agents-and-automation/inspect-evaluation-harness-starter/examples/toy_eval_cases.json \
--summary-out scratch/agents/inspect_evaluation_harness_summary.json \
--log-dir scratch/agents/inspect-eval-logs
Notes
- This starter intentionally avoids external model APIs so it can run in the repository sandbox.
- The candidate and baseline solvers are both deterministic; the purpose is to verify the harness and comparison surface, not to benchmark large models.