Benchmark adder

benchmark-adder

You are creating a new Claude Code plugin that wraps an arbitrary benchmark repository so a user can benchmark their current harness + plugins + skills + model + MCP servers against it.

This skill is a thin orchestrator. The heavy lifting is a multi-phase, convergent, spec-driven, TDD + adversarial-review babysitter process that lives at:

.a5c/processes/benchmark-plugin-creator.js

All you do here is:

1. Collect inputs from the user (or parse them from $ARGUMENTS).
2. Prepare the inputs file.
3. Invoke the babysitter babysit skill, pointing it at the process.
4. Hand control to the babysitter orchestration.

Inputs

Field	Required	Default	Example
benchmarkRepoUrl	yes	—	https://github.com/arcprize/arc-agi or https://github.com/xiaowu0162/longmemeval
pluginName	no	derived from repo	arc-agi-benchmarker, longmemeval-benchmarker
outputDir	no	./plugins	./plugins
author	no	empty	Tal Muskal
targetHarnesses	no	["claude-code", "codex", "gemini", "opencode"]
additionalRequirements	no	empty	free-form extra requirements
autoPush	no	false	if true, the integration phase runs git push after committing the new plugin

Parse $ARGUMENTS as benchmarkRepoUrl plus optional key=value tokens. If benchmarkRepoUrl is missing, ASK the user for it before proceeding.

Steps

Step 1 — Verify prerequisites

Check that the babysitter SDK is available and jq is installed. If not, tell the user to install them (the babysitter skill SKILL.md has the exact commands) and stop.

command -v jq >/dev/null 2>&1 || { echo "Install jq first."; exit 1; }
command -v babysitter >/dev/null 2>&1 || echo "babysitter CLI not found — the babysit skill will install it."

Step 2 — Verify the process file exists

ls .a5c/processes/benchmark-plugin-creator.js >/dev/null || {
  echo "Missing .a5c/processes/benchmark-plugin-creator.js"
  exit 1
}

If it is missing, stop and report to the user — the skill cannot run without it.

Step 3 — Write the inputs file

Write the inputs next to the process so babysitter can pick them up:

mkdir -p .a5c/processes
cat > .a5c/processes/benchmark-adder.inputs.json <<'JSON'
{
  "benchmarkRepoUrl": "<BENCHMARK_REPO_URL>",
  "pluginName": "<PLUGIN_NAME_OR_EMPTY>",
  "outputDir": "./plugins",
  "author": "<AUTHOR_OR_EMPTY>",
  "targetHarnesses": ["claude-code", "codex", "gemini", "opencode"],
  "additionalRequirements": "<FREE_FORM_OR_EMPTY>",
  "autoPush": false
}
JSON

Replace the placeholders with actual values parsed from $ARGUMENTS or collected from the user. Leave optional fields as empty strings if not provided — the process has defaults.

Step 4 — Invoke the babysitter:babysit skill

Use the Skill tool with skill: "babysitter:babysit". Pass arguments pointing at the process and inputs:

skill: babysitter:babysit
args: --process-file .a5c/processes/benchmark-plugin-creator.js \
      --entry .a5c/processes/benchmark-plugin-creator.js#process \
      --inputs .a5c/processes/benchmark-adder.inputs.json \
      --harness claude-code

The babysitter skill will:

• install / verify the SDK at the repo-pinned version
• create a run directory under .a5c/runs/<runId>/
• iterate through the benchmark-plugin-creator phases
• surface breakpoints for user review (benchmark classification, architecture, etc.)
• produce the plugin in ./plugins/<pluginName>/ plus a spec folder alongside it
• register the new plugin in .claude-plugin/marketplace.json
• update the top-level repo README.md to list the new plugin
• git add + git commit the new plugin, marketplace entry, and README update (and git push if autoPush=true)

Step 5 — Report back

Once babysitter finishes (or yields a breakpoint), report to the user:

• run id and .a5c/runs/<runId>/ path
• path to the generated plugin
• final convergence score and ship-ready flag
• any remaining tech debt from the final adversarial review

Notes

• This skill does NOT run the generated benchmark plugin. It only creates it. After creation, the user installs it as a normal Claude Code plugin and runs its own /setup and /run-benchmark skills.
• The process is designed to converge to >=99% spec parity per phase, with 3 converging parts per phase (plan, build, refine) and an adversarial review that can include online research.
• Supported benchmark families: agentic-env, memory-eval, dataset-qa, code-eval, tool-use, generic. The process auto-detects and branches accordingly in Phase 4.
• Reference for plugin layout: specializations/meta/plugin-creation in the babysitter process library, and the existing plugins/arc-agi-benchmarker/ in this repo.

Example invocations

/benchmark-adder https://github.com/arcprize/arc-agi
/benchmark-adder https://github.com/xiaowu0162/longmemeval pluginName=longmemeval-benchmarker
/benchmark-adder https://github.com/openai/human-eval author="Tal Muskal"