Benchmark models

benchmark-models

Run the same prompt across providers and compare them on data, not vibes. The deterministic work lives in scripts/model_benchmark.py (stdlib-only, ported from gstack's bun benchmark; no SDKs, no gstack runtime). This skill is the interactive wrapper: pick a prompt, preview auth with a dry-run, confirm providers, run, interpret, optionally save.

Providers wrap existing CLIs and their own auth (no API keys needed except the optional judge):

• claude via claude -p --output-format json
• gpt via codex exec ... -s read-only --json
• gemini via gemini -p ... --output-format stream-json

Triggers

Invoke this skill when the user says any of:

• benchmark models
• compare models
• model shootout
• which model is best
• cross-model comparison

Do NOT invoke for web page performance (that is a different concern).

Preconditions

• At least one provider CLI installed and authenticated. The Step 1 dry-run reports exactly which are ready; if none are, the skill stops.
• The optional quality judge needs ANTHROPIC_API_KEY (adds ~$0.05/run).

Process

Resolve the driver first. In a repository checkout, use BENCH="$(git rev-parse --show-toplevel)/.claude/skills/benchmark-models/scripts/model_benchmark.py". From an installed skill copy, use the local skill path: BENCH="$PWD/scripts/model_benchmark.py" when your shell is in the benchmark-models skill directory.

Step 1: Pick a prompt and preview auth (dry-run)

Decide the prompt with AskUserQuestion:

• A) Benchmark a skill: pass that skill's SKILL.md path as the prompt. To list candidates, search the installed skills dir, not just . (skills usually live under ~/.claude/skills/ or ~/.copilot/skills/, not the project root).

  Allowed-root constraint: $BENCH reads a prompt file only when it resolves under the repo checkout, the current directory, or a path listed in MODEL_BENCHMARK_SKILL_ROOTS (see resolve_prompt and _allowed_prompt_roots in the benchmark script). A SKILL.md under ~/.claude/skills/ or ~/.copilot/skills/ is outside the repo and cwd, so the run aborts with exit code 2 ("prompt file must be under an allowed root") and benchmarks nothing. Before running, export the skills dir, for example export MODEL_BENCHMARK_SKILL_ROOTS=~/.claude/skills (or ~/.copilot/skills), or copy the file into and run from an allowed root. A prompt already inside the repo or cwd needs no env var.

• B) Inline prompt: pass --prompt "<text>".

• C) A prompt file on disk: pass its path (verify it exists first).

Then preview availability without spending anything:

python3 "$BENCH" --prompt "dry-run" --models claude,gpt,gemini --dry-run

The "Adapter availability" block shows OK or NOT READY (with a remediation hint) per provider. If ALL are NOT READY, STOP: nothing can run. Suggest claude, codex login, or gemini login / export GOOGLE_API_KEY.

Step 2: Confirm providers and judge

From the dry-run, AskUserQuestion which providers to include (default: all authed). Unauthed providers are skipped cleanly and never abort the batch.

If ANTHROPIC_API_KEY is set, AskUserQuestion whether to enable --judge (quality scoring, ~$0.05). Never auto-enable it; it costs real money. If the key is absent, omit --judge.

Step 3: Run

python3 "$BENCH" <prompt-spec> --models <picked> [--judge] --output table

<prompt-spec> is --prompt "<text>" (B) or a file path (A/C). Stream the output; each provider runs the full prompt, so expect 30s-5min.

For an installed-skill path (A) under ~/.claude/skills/ or ~/.copilot/skills/ (outside the repo and cwd), export MODEL_BENCHMARK_SKILL_ROOTS first (see Step 1 Option A) or the run aborts with exit code 2 before any provider starts.

Step 4: Interpret and optionally save

Summarize: fastest (latency), cheapest (cost), highest quality (if judged), and a best-overall call that names the tradeoff. Call out any provider that errored, with its remediation path. Offer to save a JSON baseline so future runs can be diffed for regressions:

python3 "$BENCH" <prompt-spec> --models <picked> [--judge] --output json > benchmark-$(date +%Y%m%d)-<slug>.json

Scripts

model_benchmark.py

Runs the benchmark. Pure stdlib; safe to run directly.

python3 scripts/model_benchmark.py <prompt-file | --prompt "text"> \
  [--models claude,gpt,gemini] [--judge] [--dry-run] \
  [--output table|json|markdown] [--workdir PATH] [--timeout-ms N] [--skip-unavailable]

Exit codes (ADR-035): 0 success, 1 logic/runtime error, 2 config or usage error, 3 external dependency failure, 4 authentication or authorization failure.

Notes:

• --dry-run resolves auth and prints availability without sending any prompt.
• The judge posts to the Anthropic Messages API via stdlib http.client (no SDK) and fails soft: a judge error warns and leaves quality blank, it never aborts the run.
• Do not place secrets in prompts. The gpt and gemini CLIs receive the prompt through command arguments, which some systems expose through process listings or crash reports.
• Safety asymmetry: the gpt/codex adapter runs -s read-only; the gemini adapter passes --yolo (auto-approve), which is NOT sandboxed. Benchmark in a disposable --workdir if the prompt could trigger file writes.
• The driver only runs providers concurrently when every selected provider is read-only GPT/Codex. Claude and Gemini run sequentially because they can mutate the workdir.
• Pricing lives in the PRICING table at the top of the script. An unpriced model costs 0.0 with a stderr warning; add a row rather than guess.

Anti-Patterns

Avoid	Why	Instead
Running the benchmark before the dry-run	Spends API calls on unauthed providers that will just error	Always Step 1 dry-run first; only run authed providers
Auto-enabling --judge	It adds real per-run cost the user did not approve	Opt-in only, after the Step 2 question
Benchmarking in a live repo with a write-prone prompt	The gemini adapter is not sandboxed (--yolo)	Use a disposable --workdir
Guessing a price for an unlisted model	Produces a wrong cost number that looks authoritative	Add a row to the PRICING table; unlisted stays 0.0 with a warning
Listing skills to benchmark via find . only	Installed skills live under ~/.claude / ~/.copilot, not the project root	Search the installed skills dir

Verification

The skill run is complete when:

• Step 1 dry-run ran and its availability block was shown to the user.
• If zero providers were authed, the run STOPPED (no benchmark attempted).
• The benchmark ran only the user-confirmed providers.
• If the prompt was a skill SKILL.md outside the repo/cwd, MODEL_BENCHMARK_SKILL_ROOTS was set (or the file staged under an allowed root) so the run did not abort with exit code 2.
• --judge was included only after explicit user opt-in.
• Results name the fastest, cheapest, and (if judged) highest-quality model, with errors and their remediation surfaced.
• python3 -m pytest .claude/skills/benchmark-models/tests passes (the driver carries unit tests at 100% coverage).