agentsclimarketplace

Floom skill evals

Skill floomhq/starter/skills/floom-skill-evals

Floom Starter Pack. 65 hand-picked AI agent skills for Claude Code, Codex, Cursor, Kimi, OpenCode. One install, auto-activates. +16.2pp pass-rate lift.

Install
npx -y skills add floomhq/starter --skill floom-skill-evals

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use when the user asks to create, run, or review skill evaluations with baseline-vs-with-skill A/B tests, Docker-isolated task runs, deterministic verifiers, receipt generation, or claims like pass-rate lift for a skill or pack.

SKILL.md

2.4 KB, as published. Nobody here has run it

Floom Skill Evals

Run simple, deterministic skill receipts:

  1. Create eval tasks for a skill.
  2. Run baseline and with-skill conditions in isolated Docker workspaces.
  3. Grade with deterministic verifier.
  4. Save result.json and a markdown receipt.

When to use

Use this skill for requests like:

  • "evaluate this skill"
  • "baseline vs with skill"
  • "run skill receipts"
  • "prove skill lift"
  • "Floom Verified style run"

Commands

1) Scaffold an eval task

python3 scripts/scaffold_eval_task.py \
  --skill seo-audit \
  --task technical-homepage-audit \
  --out ./skills/seo-audit/evals

This creates:

  • <out>/<task>/task.md
  • <out>/<task>/input/
  • <out>/<task>/expected.json
  • <out>/<task>/verifier.py
  • <out>/<task>/eval.json

2) Run A/B eval

python3 scripts/run_skill_eval.py \
  --eval-dir ./skills/seo-audit/evals/technical-homepage-audit \
  --skill-path ~/.codex/skills/seo-audit \
  --agent codex

Outputs:

  • runs/<timestamp>/result.json
  • runs/<timestamp>/report.md
  • condition workspaces and logs

Default policy rejects scripted output commands (echo, printf, heredoc writes) in agent_command. Use --allow-scripted only for runner plumbing tests.

Execution model

  • baseline: run agent command without skill mount.
  • with_skill: run same command with skill mounted at /skill and env FLOOM_SKILL_PATH=/skill.
  • verifier: runs deterministic checker inside each condition workspace.
  • audit: simple deterministic summary from pass/fail/runtime deltas.

Required eval contract

eval.json must include:

  • id
  • skill
  • timeout_minutes
  • trials
  • agent
  • agent_command.baseline
  • agent_command.with_skill
  • verifier
  • expected_output

Command placeholders supported:

  • {task_dir} mounted as /task
  • {output_path} path in workspace where agent writes result

Notes

  • This v0 runner is Docker-first and local-first.
  • Host keeps credentials; container only gets mounted task workspace.
  • If API keys are needed, pass scoped env explicitly with --pass-env OPENAI_API_KEY etc.
  • Receipts include provenance hashes for verifier and both condition commands.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.