Blender agent benchmark

Skill ifBars/blender-agent-studio/plugins/blender-agent-studio/skills/blender-agent-benchmark

Codex plugin for reproducible Blender modeling, validation, animation, MCP tooling, and agent benchmarking

Install

npx -y skills add ifBars/blender-agent-studio --skill blender-agent-benchmark

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

12 days oldThe repository was created 12 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Benchmark Blender modeling agents, skills, prompts, scripts, or MCP tools with paired isolated runs. Use for baseline-versus-plugin comparisons, regression suites, skill forward-testing, MCP usefulness evaluation, score calibration, or claims that a Blender workflow improves mesh, visual, structural, animation, cost, or completion quality.

SKILL.md

3.4 KB, as published. Nobody here has run it

Blender Agent Benchmark

Measure changes with the same tasks, model, effort, limits, Blender build, and evaluator. Preserve natural agent behavior.

Protect benchmark integrity

Create isolated directories for every condition and repetition.
Do not leave the other condition's code, renders, metrics, or expected fixes where the agent can discover them.
Keep the user-facing task prompt identical except for explicit skill invocation in the plugin condition.
Use codex exec --ignore-user-config for the no-plugin baseline.
Use the installed plugin in a fresh invocation for the plugin condition.
Record CLI version, model, effort, Blender build, duration, tool calls, failures, and output hashes.
Evaluate outputs after generation. Do not leak hidden rubric details to the agent.

Read references/methodology.md before changing fixtures, scoring, or comparison claims. Read references/validated-results.md only when reviewing the plugin's recorded validation result, not while generating a benchmark submission.

Run the suites

Use scripts/run_benchmark.ts:

bun "<skill-root>\scripts\run_benchmark.ts" `
  --suite quick `
  --mode baseline `
  --output "<run-root>\baseline"

bun "<skill-root>\scripts\run_benchmark.ts" `
  --suite quick `
  --mode plugin `
  --output "<run-root>\plugin"

Start with a smoke task to validate the harness. Use at least three representative tasks and repeated runs before claiming a general capability gain.

Use scripts/compare_runs.ts for randomized blinded multiview judging. Use scripts/rescore_run.ts to recompute deterministic scores after a scorer change without rerunning agents.

Noninteractive Codex cancels MCP tool calls that require approval. If testing an MCP condition, pass --bypass-approvals to every compared condition and use isolated benchmark directories. Do not give only the MCP condition broader permissions.

Use scripts/benchmark_mcp.ts --asset <path> --output <new-dir> to verify that the MCP transport returns the same deterministic metrics as direct CLI evaluation. Equivalent results prove transport correctness, not a modeling quality gain.

Compare conditions

Keep these dimensions separate:

execution/export validity;
prompt and structural compliance;
geometry/game-readiness;
multiview visual quality;
physical plausibility;
animation quality when applicable;
context/export correctness;
time, turns, tool failures, and cost.

Use hard gates before the weighted score. Prefer blinded pairwise visual review over uncalibrated absolute aesthetic scores.

Iterate

Inspect failed metrics, renders, videos, and traces.
Identify one transferable workflow defect.
Change the smallest relevant skill, deterministic tool, or MCP surface.
Rerun the same slice.
Run a holdout task before retaining the change.

Do not retain benchmark-specific instructions that reveal fixture answers or damage ordinary modeling behavior.