Marshal benchmark

Skill chiruu12/marshal/skills/marshal-benchmark

Compare AI coding strategies on a real task with Marshal: run the same goal through several configured clients (e.g. a premium model vs a cheaper one) and get a measured cost/latency/outcome comparison. Use when you want evidence for which provider/model to route a kind of work to, or to prove what a cheaper strategy actually saved. Honest by construction - it measures, it never guesses.From its SKILL.md

Install

npx -y skills add chiruu12/marshal --skill marshal-benchmark

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

2.8 KB, as published. Nobody here has run it

Benchmarking routing strategies

Marshal's differentiator is proving what a routing choice costs, with measured numbers - not a fixed % claim. A strategy is a configured client (a backend + model + permission combo). This playbook runs one goal through N strategies and compares them.

Run it

list_clients - pick the strategies (client names) to compare, e.g. a premium worker vs a cheap one, both capable of the task.
benchmark(goal, clients, task_id?, max_concurrency?) - runs the same goal through each client in parallel (each in its own worktree), then returns a comparison:
- strategies[] - per client: status, cost_usd, source, duration_ms, tokens, run_id.
- cheapest / fastest - the winning client, but only among strategies that succeeded and have a known cost. A strategy whose cost is unavailable never wins "cheapest."
report(task_id) re-derives the same comparison later from the ledger (read-only) - reproducible and auditable.

Multi-workspace: benchmark and report are scoped to one workspace, and task_ids are per-workspace. If you benchmark with workspace=<name>, pass the same workspace to report(task_id, workspace=<name>) - otherwise it reads the wrong repo's ledger and finds nothing.

Read it honestly

Check source on each row. native = the backend reported the cost; admin-api = the real charge read back from a provider usage API (e.g. EastRouter); unavailable = unknown (e.g. Cursor without the Admin API, or a token-only Codex run). Never read an unavailable $0 as free.
The benchmark measures cost, latency, and outcome status - not correctness. A cheapest strategy can still produce worse code. Review the diffs (collect_run) before drawing conclusions.
empty / failed / verify_failed strategies still cost tokens; they appear in per-run cost but are excluded from cheapest / fastest.

What to do with the result

Route future work of this kind to the strategy with the best cost/quality trade-off (the cheapest one that also produces correct diffs).
To keep a strategy's output, integrate its run_id (see marshal-orchestrate).
The historical record of what each strategy cost - derived on read, never a stored guess - is the defensible asset. Build it up over real tasks rather than promising a universal savings number.