Operate benchmark lab
Skill understudylabs/understudy-agent-tools/skills/operate-benchmark-lab
Use when a coding agent must operate the full benchmark lifecycle over local benchmark dirs — "build a benchmark from my traces and run models on it", "review and calibrate the eval", "queue a prompt experiment", "is an executor running", "read the rigor report". Covers traces → build-benchmark → review/feedback → calibration floors → candidate and prompt-override runs → rigor/CI reading → app-replay regression, via the benchmarks MCP server or CLI verbs, plus the run-executor daemon lifecycle.From its SKILL.md
npx -y skills add understudylabs/understudy-agent-tools --skill operate-benchmark-labAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 18 stars18 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
15.2 KB, as published. Nobody here has run it
Operate the benchmark lab
The operator's manual for the whole benchmark/experiment lifecycle a coding
agent can drive end to end. Two interfaces over the same sidecar files:
the benchmarks MCP server (preferred for agents — same validation code as
the hub API) and the CLI verbs. Execution always happens in a separate
executor process; the MCP server and hub never run models. Command matrix,
artifact map, and daemon details in reference.md; the tool
table and agent loop in docs/agent-operator-surface.md.
Resolve CLI
Prefer the installed understudy binary. If it is unavailable inside a repo
checkout, run through the package script:
npm run build
node dist/bin.js benchmarks mcp --root <dir>
MCP registration (Claude Code ~/.claude.json → mcpServers):
{ "understudy-benchmarks": { "type": "stdio", "command": "understudy", "args": ["benchmarks", "mcp"] } }
— default root ~/.understudy/benchmarks; add --root per extra directory.
Safety Gates
- Queueing is not executing.
queue_run/understudy runs queueonly writes a request file. Model rollouts spend gateway money only when an executor picks the request up; say which executor will, before queueing. - Trust posture, not per-call dialogs. Spend-adjacent shapes (multi-arm
or multi-rollout runs, implicit all-task runs, experiment
approval/verdict patches) consult the one-time posture in
~/.understudy/trust.json(understudy trust set, levelslocal_sandbox<bounded_experiments<hosted_ops). Atbounded_experiments+ they proceed with a visible one-line notice (arm count, rough cost) — surface that notice to the user, then keep moving. Below that, the guard returns the one action to offer (understudy trust set bounded_experiments);confirm: trueafter explicit in-chat consent stays a per-call escape hatch. There is NO default spend cap: the posture'sallow_spend_usd_per_runis an opt-in generous stop-loss (warn at 1x with a recordedspend_warning; hard stop only at 2x withspend_stop). - Local arms are machine-aware. On predicted OOM (onboarding profile /
memory probe) or a serve failure, the executor runs the arm on the gateway
base model and records it —
arm_fallbackevent plusfallback_reasonon every row. Report the fallback; never present a fallen-back arm as a local measurement. - One executor per benchmark dir. Before starting
runs execute --watch, check for a live claim (claimed_byon the request,executor_versionon events) — a stale watcher built before a feature landed is the classic corruption hazard; new requests carryrequires:[...]so old executors skip them withrun_unsupportedinstead of running them bare. - Reviews and feedback are append-only ledgers; never edit
reviews.jsonl/feedback.jsonllines in place. Generated tasks are born accepted (review-policydefault_decision: "accept") —reviews.jsonlcarries explicit overrides only.apply_auto_acceptsmatters only for benchmarks opted intodefault_decision: "pending", and is itself the explicit user action — invoke it only when the developer asked. - Honest reporting only: anomaly rows (
rollout_timeout,app_replay_unobserved, structural sentinels) are excluded from aggregates but reported, never fabricated as scores. Overlapping CIs are a tie. - Verifier-only changes regrade, never rerun. When only the verifier group changed (gold, contract, rubric, metric config — a MINOR bump), re-score the existing trajectories; queueing fresh rollouts for a verifier fix wastes gateway money and destroys comparability. Rerun is reserved for env-group (MAJOR) changes.
- Fixture-test every verifier before trusting it. One known-valid result must pass and one plausible-but-wrong result must fail. A verifier that has never rejected a wrong answer has not been tested; do not regrade or calibrate against it.
Outcome-first replacement loop
When the requested result is “meet or beat the incumbent” rather than merely “run this model,” operate one hash-bound decision loop:
- Bind each task, history, trajectory, final effects, and outcome contract to one authoritative source execution. Do not optimize ambiguous or reconstructed rows.
- Calibrate the verifier with a known-valid oracle and plausible wrong sentinels, then calibrate difficulty to prove measurable dev headroom.
- Freeze immutable train, dev, and holdout boundaries. Optimize on train, make method decisions from canonical dev receipts, and leave holdout untouched until an explicitly authorized promotion evaluation.
- Measure the incumbent and candidate through the same canonical serving contract. Preserve per-family scores, cost, latency, failures, and hashes; aggregate-only evidence is insufficient for regression gates.
- Run GEPA only after those gates pass. Feed its canonical dev receipt into the fail-closed method ladder to decide whether to continue GEPA or advance to SFT, DPO, or GRPO within the remaining budget.
- Before any promotion claim, require serving parity, strict arm evidence,
protected-family no-regression, immutable artifacts, and an authorized
fresh holdout receipt.
target_meton dev is a candidate-selection result, not production evidence.
The exact contracts, failure boundaries, and current executor integration
status are in references/outcome-first-replacement-loop.md.
The method-ladder decision contract is in
references/method-ladder.md.
The lifecycle
- Intake (dropped workloads). When the user drops a folder or file into
the chat (or names a local path),
profile_workloadscans it locally (payloads unread, nothing leaves the machine) and lists foundry-consumable dataset candidates. Discuss what the workload is, confirm the dataset and label/input columns with the user, thenfrom_datasetcompiles it into a PROPOSED benchmark (review-pending,executable: false) whose tasks land in the review inbox. Never queue a run from an intake — the user reviews the proposal first (step 2). 0.5. Interview before bulk authoring. Never batch-author tasks straight from an intake or trace corpus. First, from the trace clusters, propose 2–3 capability directions, each as a card: Name (the capability being measured), Example (one concrete task drawn from the traces), Tests (what the verifier would check), Needs (dependencies — services, fixtures, tools the environment must supply). The user picks a direction, then approves a posture per dependency:live(real service; usually wrong for a benchmark),frozen(recorded responses), orsimulated(seeded synthetic implementation). Then build one task first: run it end to end and audit both trajectories — the agent's (did the environment answer sensibly, did the task make sense) and the verifier's (did scoring reflect what actually happened). Fixture-test the verifier (one valid pass, one plausible-wrong fail — see Safety Gates). Only after that pilot task survives audit do you batch-author the rest. - Build. Captured traces →
understudy traces build-benchmark, thenunderstudy traces author-tasks(gateway) for legible task definitions. The generated environment already ships the fixtures pre/post split (candidate-readablefixtures.jsonvs scorer-onlygold.json) and the tiered gold-leakage audit (tier 1 verbatim findings, tier 2 fuzzy advisory) recorded inmanifest.leakage_audit— read it, don't re-derive. - Review (born accepted).
list_benchmarks→read_benchmark→read_task. Generated tasks are accepted by default — nothing to apply.read_benchmarkreturns each task'seffective_decisionplusattention_flags(low confidence, self-check, incumbent-failed, schema conflict, anomaly): work the flagged tasks, tweaking viasubmit_feedbackor overriding viasubmit_review(reject | needs_more | restrict, oracceptto re-accept an overridden task; newest line per task wins).apply_auto_acceptsexists only for benchmarks runningreview-policy.jsondefault_decision: "pending"(the old explicit-accept flow). Task or environment wrong?submit_feedbackappends the ledger entry and returns the regenerate-env handoff (understudy traces regenerate-env). - Promote and calibrate. Gate promotion on
understudy benchmarks rigor <dir> --ci(nonzero exit = findings to fix first), thenunderstudy traces promoteunlocks full runs. Queue the incumbent (--incumbent/incumbent_models) plustrivial_arms: ["null_agent", "spam_agent"]. Readcalibration.json:null_floor/spam_floorwithfloor_exceeded: true(> 5%) names thepassed_task_idsto fix; incumbent-failed tasks are flagged suspect — fix the benchmark before trusting any candidate score. - Run candidates and prompt experiments.
queue_runwith candidate models; prompt experiments rideprompt_overrides(--prompt-override <arm_label>=<model>=<suffix-file>, e.g. an SOP suffix appended to each task's system prompt) so prompt and model arms land in the same run under distinct arm labels. - Execute. Make sure exactly one executor is running:
understudy runs execute --benchmark <dir> --watch(daemon; polls every 30s) or omit--watchfor a single pass. Pollrun_statusuntil rows land;rollout_timeoutkills hung rollouts into anomaly rows. - Read results rigorously.
read_rollout/diff_rollouts(tool_sequence.diverges_at, unmet obligations), thenunderstudy benchmarks rigor <dir>→rigor-report.md— the ABC attestation (oracle solvability, floors, calibration, contract complexity, anomalies, contamination provenance; honest UNKNOWN rows for unchecked items). Bootstrap CIs perdocs/benchmark-rigor.md. - Regression after code edits. When the fix was to the user's app, run
the
app_replayarm via../replay-app-harness/SKILL.md— same frozen tasks, current code, rows never feed calibration.
Versioning lifecycle (rerun / regrade / reuse)
Every task carries an optional semver version plus content_hashes
(env_sha256, verifier_sha256, meta_sha256) — three canonical-JSON
sha256 hashes over the field groups defined in docs/benchmark-rigor.md.
The contract (mirrors Harbor's model):
| Changed group | Bump | Consequence |
|---|---|---|
| env (instruction, fixtures, tool surface, seed, unknown fields) | MAJOR | rerun — old trajectories are invalid |
| verifier (gold refs, contract/rubric, metric config) | MINOR | regrade — trajectories stand, re-score them |
| meta (title, docs, tags) | PATCH | reuse — existing rows as-is |
Benchmark-level version is the max bump across tasks (added tasks = MAJOR,
removed = MINOR). Every bump appends one
understudy.benchmark_version.v1 line to versions.jsonl (append-only
sidecar; records the bump, per-task reasons, splits hash, contamination
status).
Operate it with three verbs:
understudy runs regrade --benchmark <dir> [--run <id>] [--task <id>...] [--dry-run]— verifier-only change: re-scores retained trajectory evidence offline against the CURRENT verifier and writes fresh rows under<run>-regrade-<n>(source-run provenance, original cost/latency preserved), then appends one MINORversions.jsonlline for the regraded tasks so the superseded source rows go stale instead of double-counting next to their regrades. Never queues rollouts, never spends gateway money. If you're about to rerun after a gold/rubric fix, stop — regrade instead.understudy benchmarks upgrade <dir> --against <old-benchmark.json>— diffs the current manifest against the archived previous one and prints the minimal plan first: which task ids rerun, which regrade, which reuse, and the resulting version bump; then appends oneversions.jsonlline (--dry-runskips the append). Nothing executes;--queue --model <id>writes run requests for the rerun set only.understudy benchmarks rigor <dir>... --ci— the gate beforeunderstudy traces promoteand before publishing any new version: exits nonzero on hard findings (floors exceeded, oracle unsolved, verbatim leakage, contamination) instead of just writing the report; UNKNOWN is fatal only with--strict. Run it in the upgrade path too — a version bump that fails rigor does not promote (traces promoteenforces this;--override-rigor <reason>is recorded in the promotion record).
Leaderboard staleness. A row is stale when it predates the newest
breaking bump for its task (MAJOR or MINOR line in versions.jsonl; rows
without created_at are conservatively stale). MAJOR means stale —
rerun (the trajectory itself is invalid); MINOR means stale — regrade
(trajectory reusable, score not); PATCH-only changes leave rows current.
Report stale rows the honest-reporting way: excluded from headline
aggregates, counted and named, never silently dropped or silently included.
Daemon lifecycle (executor + desktop)
- Check first:
ps/events for a liveruns executewatcher; the first stderr line printsexecutor version <v> (pid <n>). A live foreign claim is respected; a dead claim is taken over (staleness takeover). - Start:
understudy runs execute --benchmark <dir> --watchin the background; single-pass (no--watch) for one drain. Stop safely by killing the watcher between polls — requests are re-claimable; in-flight rollouts past the timeout become anomaly rows, never silent hangs. - Desktop app daemon (separate thing — local model serving, not the run
queue):
understudy daemon statuspid-checks and health-probes~/.understudy/agent-card.json; thenunderstudy desktop capabilities/status/model|slot|download|chatverbs. Seereference.mdfor what the app covers vs headless CLI.
Output Standard
End with: benchmark dir and stage (proposed/promoted); current benchmark
version (semver, from the manifest / last versions.jsonl line) and
stale-row status (rows counted by current / stale-regrade / stale-rerun,
and whether any stale rows sit in a headline aggregate); ledger state
(reviews, auto-accepts, feedback); calibration verdict (incumbent gate,
null/spam floors, suspects); runs queued/executed and by which executor
version; result type (validation, oracle, live, app-replay); artifact paths
written (calibration.json, rigor-report.md, rows-*.jsonl,
versions.jsonl); and one recommended next command.