agentsclimarketplace

Model bench

Skill nateslabach/skills/skills/model-bench

Run a 3-leg evaluation against a newly released LLM. Leg 1 is trap questions with deterministic answers, leg 2 is an agentic orchestration task with a planted failure, leg 3 is a brownfield feature build in a real repo. Use when the user wants to vibe-check, test, evaluate, or compare a new frontier model, or mentions model-bench, model vibe check, or launch-day protocol.From its SKILL.md

Install
npx -y skills add nateslabach/skills --skill model-bench

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • reads credentialsReads from 2 credential sources: `~/proctor/model-bench/answer-key.md` and 1 more.
  • 6 stars6 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

4.7 KB, ~1.1k tokens by cl100k_base, as published. Nobody here has run it

model-bench

3 legs, each aimed at a different failure mode:

  1. Trap questions: does it think, or pattern-match to training data? Deterministic answers, grades in seconds.
  2. Orchestration: can it plan, delegate, and recover when a step fails (including a planted trap)?
  3. Brownfield build: can it ship a feature into an existing codebase without breaking things 2 files away?

Each leg scores 0-5. Total out of 15.

Contamination rules (read first)

  • The model under test must NEVER see gold answers or learn about the planted trap. If it does, that leg is void.
  • Private answer key lives outside this repo: ~/proctor/model-bench/answer-key.md (or grading/answer-key.local.md, gitignored). Public stub: grading/answer-key.md. Legs 2–3 checklists: grading/rubrics.md.
  • This skill runs in a proctor session (Nate plus a model he already trusts). The model under test only ever receives the task prompts — never this skill folder, never the private key.
  • Never use this repo as the target repo for legs 2–3.
  • Leg 1 isolation: launch each question in a context with no access to this workspace (no tools / empty workspace / chat-only). If the target model can Read files here, the leg is void.
  • One fresh session per leg. No custom system prompts or rules that would coach the model.

Workflow

Progress:
- [ ] Setup: pick target model, pick target repo, verify leg 2 plant
- [ ] Leg 1: proctor launches 5 parallel subagents (one question each), record final answers
- [ ] Leg 2: paste orchestration task into a fresh agent session on the target repo
- [ ] Leg 3: paste brownfield spec into a fresh agent session, branch per model
- [ ] Grade leg 1 with the private answer key; legs 2–3 with grading/rubrics.md
- [ ] Fill scorecard, save to results/, write 2-3 sentence verdict per model

Setup

  • Target repo for legs 2-3: a real repo with existing conventions, ideally one Nate knows well enough to judge output quality. Never this repo.
  • Verify the leg 2 planted path (default docs/METRICS.md) does not exist in the target repo. If it does, pick another plausible-but-missing path and note it.
  • When comparing models, use identical prompts, identical repo state, and a fresh branch per model (vibe/<model-name>).

Run the legs

Leg 1 (proctor-driven) The proctor AI runs leg 1. Do not make the human paste questions. Launch 5 parallel subagents with the target model slug, each receiving exactly one fenced question from tasks/leg-1.md and nothing else. Record final answers only. Isolation: those subagents must not be able to read this repo (no tools, or empty/disposable workspace). Same-workspace leg 1 = void. If the target model has no subagent slug (common on launch day), use five fresh chats the same way — still proctor-driven, not human paste. Do not use subagents for legs 2–3 in this workspace. Those legs run in separate sessions on the target repo.

Grade

  • Leg 1 gold: private key at ~/proctor/model-bench/answer-key.md (see grading/answer-key.md stub)
  • Legs 2–3: grading/rubrics.md
  • Only open the private key after all legs have been run, and never in a session where the target model is active.

Results

Save each filled scorecard to results/YYYY-MM-DD.md (append -2, -3 for multiple runs on one day). Use this template:

# Vibe check: [date]

Target repo: [repo]
Leg 2 plant: [path used]
Leg 3 spec: [feature, one line] / planted ambiguity: [what it was]

| Leg | [Model A] | [Model B] | [Model C] |
|---|---|---|---|
| 1. Trap questions (/5) | | | |
| 2. Orchestration (/5) | | | |
| 3. Brownfield (/5) | | | |
| **Total (/15)** | | | |

Verdict per model: which legs it won, what surprised me, does it earn a spot in the rotation.

Notes

  • Public trap questions decay. When a question gets 3 passes in a row across model families, retire it and add a fresh one to the bank.
  • Different models winning different legs is the expected (and most interesting) result. Record it, it's post material.

What ships with it: 6 files

8.5 KB alongside SKILL.md

grading/

results/

tasks/

Keep looking

Skills are one crate of 611,676. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.