agentsclimarketplace

Ai agent bench

Skill reidemeister94/development-skills/skills/ai-agent-bench

Compare Claude Code and Codex on the same real code-change task with isolated worktrees, identical gates, transcripts, time, and cost.From its SKILL.md

Install
npx -y skills add reidemeister94/development-skills --skill ai-agent-bench

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 12 stars12 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.7 KB, as published. Nobody here has run it

AI agent bench

Compare agents only with the same task, starting commit, and outcome check. The harness preserves result branches and removes temporary worktrees.

Create <repo>/.agent-bench.toml:

prompt = "prompts/task.md"
start_branch = "main"              # or start_commit
agents = ["claude", "codex"]
outer_check = "./scripts/full_check.sh"
inner_check = "pytest tests/integration/test_x.py -q"

outer_check proves the real outcome before and after, and measures wall time. inner_check gives agents fast feedback.

Require a clean repo, available CLIs, and a passing outer_check. Confirm agents and run ID, then run trials sequentially to avoid load-biased timing:

python <skill>/scripts/run_trial.py --repo "$REPO" --config "$REPO/.agent-bench.toml" --agent "$AGENT" --run "$RUN_ID"

Results go to eval-results/<task>/<agent>/run-<id>-<timestamp>/. Record unexpected behavior in ai-agent-bench-anomalies.md per anomalies.

Aggregate with scripts/parse_transcript.py --aggregate <run-dirs> --output comparison.json --render-report comparison.md. Report gates, branches, time delta, tokens, and cost. Never rank a failed trial.

For plugin behavior rather than a real code task, use the bounded Pydantic runner documented by eval-regression and scripts/run_evals.py.

Never commit on the user's branch. A repeated run creates a new timestamped result and preserves prior evidence.

Keep looking

Skills are one crate of 611,178. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.