agentsclimarketplace
Sign in

Floom skill evals

Skill floomhq/starter/skills/floom-skill-evals

Use when the user asks to create, run, or review skill evaluations with baseline-vs-with-skill A/B tests, Docker-isolated task runs, deterministic verifiers, receipt generation, or claims like pass-rate lift for a skill or pack.From its SKILL.md

Install
npx -y skills add floomhq/starter --skill floom-skill-evals

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

3 things to look at

  • reads credentialsReads from 1 credential source: `OPENAI_API_KEY`.
  • 3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
  • runs commandsInstructs the agent to run 2 commands, including `python3 scripts/scaffold_eval_task.py` and 1 more.

SKILL.md

2.4 KB, 549 tokens by cl100k_base, as published. Nobody here has run it

Floom Skill Evals

Run simple, deterministic skill receipts:

  1. Create eval tasks for a skill.
  2. Run baseline and with-skill conditions in isolated Docker workspaces.
  3. Grade with deterministic verifier.
  4. Save result.json and a markdown receipt.

When to use

Use this skill for requests like:

  • "evaluate this skill"
  • "baseline vs with skill"
  • "run skill receipts"
  • "prove skill lift"
  • "Floom Verified style run"

Commands

1) Scaffold an eval task

python3 scripts/scaffold_eval_task.py \
  --skill seo-audit \
  --task technical-homepage-audit \
  --out ./skills/seo-audit/evals

This creates:

  • <out>/<task>/task.md
  • <out>/<task>/input/
  • <out>/<task>/expected.json
  • <out>/<task>/verifier.py
  • <out>/<task>/eval.json

2) Run A/B eval

python3 scripts/run_skill_eval.py \
  --eval-dir ./skills/seo-audit/evals/technical-homepage-audit \
  --skill-path ~/.codex/skills/seo-audit \
  --agent codex

Outputs:

  • runs/<timestamp>/result.json
  • runs/<timestamp>/report.md
  • condition workspaces and logs

Default policy rejects scripted output commands (echo, printf, heredoc writes) in agent_command. Use --allow-scripted only for runner plumbing tests.

Execution model

  • baseline: run agent command without skill mount.
  • with_skill: run same command with skill mounted at /skill and env FLOOM_SKILL_PATH=/skill.
  • verifier: runs deterministic checker inside each condition workspace.
  • audit: simple deterministic summary from pass/fail/runtime deltas.

Required eval contract

eval.json must include:

  • id
  • skill
  • timeout_minutes
  • trials
  • agent
  • agent_command.baseline
  • agent_command.with_skill
  • verifier
  • expected_output

Command placeholders supported:

  • {task_dir} mounted as /task
  • {output_path} path in workspace where agent writes result

Notes

  • This v0 runner is Docker-first and local-first.
  • Host keeps credentials; container only gets mounted task workspace.
  • If API keys are needed, pass scoped env explicitly with --pass-env OPENAI_API_KEY etc.
  • Receipts include provenance hashes for verifier and both condition commands.

What ships with it: 31 files

24.8 KB alongside SKILL.md, 6 of them executable

references/

scripts/

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.