Ai agent bench
Skill reidemeister94/development-skills/skills/ai-agent-bench
Compare Claude Code and Codex on the same real code-change task with isolated worktrees, identical gates, transcripts, time, and cost.From its SKILL.md
npx -y skills add reidemeister94/development-skills --skill ai-agent-benchAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 12 stars12 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
1.7 KB, as published. Nobody here has run it
AI agent bench
Compare agents only with the same task, starting commit, and outcome check. The harness preserves result branches and removes temporary worktrees.
Create <repo>/.agent-bench.toml:
prompt = "prompts/task.md"
start_branch = "main" # or start_commit
agents = ["claude", "codex"]
outer_check = "./scripts/full_check.sh"
inner_check = "pytest tests/integration/test_x.py -q"
outer_check proves the real outcome before and after, and measures wall time. inner_check gives agents fast feedback.
Require a clean repo, available CLIs, and a passing outer_check. Confirm agents and run ID, then run trials sequentially to avoid load-biased timing:
python <skill>/scripts/run_trial.py --repo "$REPO" --config "$REPO/.agent-bench.toml" --agent "$AGENT" --run "$RUN_ID"
Results go to eval-results/<task>/<agent>/run-<id>-<timestamp>/. Record unexpected behavior in ai-agent-bench-anomalies.md per anomalies.
Aggregate with scripts/parse_transcript.py --aggregate <run-dirs> --output comparison.json --render-report comparison.md. Report gates, branches, time delta, tokens, and cost. Never rank a failed trial.
For plugin behavior rather than a real code task, use the bounded Pydantic runner documented by eval-regression and scripts/run_evals.py.
Never commit on the user's branch. A repeated run creates a new timestamped result and preserves prior evidence.