Evaluate with braintrust
Skill hiteshbandhu/skills-i-use/skills/ai-engineer-talks/evaluate-with-braintrust
Evaluates LLM and agent products with Braintrust—offline/online evals, SDK/CI workflows, eval platform design, Zapier/Notion product ops, and Loop-style optimization. Use when the user mentions Braintrust, eval playground, autoevals, production log evals, or agent quality platforms.From its SKILL.md
npx -y skills add hiteshbandhu/skills-i-use --skill evaluate-with-braintrustAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
- runs commandsInstructs the agent to run 2 commands, including `cp -r skills/evaluate-with-braintrust ~/.cursor/skills/` and 1 more.
SKILL.md
1.9 KB, 428 tokens by cl100k_base, as published. Nobody here has run it
Evaluate with Braintrust
Action playbook from seven Braintrust @ AI Engineer talks. Do not summarize — pick a workflow.
Supporting files: workflows.md · source-index.md
Related skill: run-llm-evals (cross-vendor eval theory).
Optional: ./skill-outputs/evaluate-with-braintrust/
Step 0 — Pick workflow
What is the user trying to do?
├─ Eval fundamentals (datasets, scorers, modes) → evals-101
├─ Build/buy eval platform architecture → platform-design
├─ Five lessons / future of evals (velocity, Loop) → eval-ops
├─ Zapier-style product + eval integration → product-ops
├─ Notion-class world-class AI quality bar → world-class-products
└─ Complex app workshop (Trainline patterns) → complex-apps
Install
cp -r skills/evaluate-with-braintrust ~/.cursor/skills/
cp -r skills/evaluate-with-braintrust ~/.codex/skills/
Source: playlists/braintrust-ai-engineer/.
Cross-cutting rules
| Rule | Source |
|---|---|
| Eval = data + task + scorers; offline and online | [src-001 @ 0:07:12] |
| Prod logs must feed datasets and online evals | [src-001 @ 0:13:13] |
| Unify offline experiments with trace replay | [src-002 @ 0:14:15] |
| 24h model swaps when evals healthy (Notion bar) | [src-003 @ 0:40] |
| Human-in-loop on Loop optimizations | [src-004 @ 0:03:33] |
Output
Name workflow; artifacts under ./skill-outputs/evaluate-with-braintrust/ when requested.
What ships with it: 3 files
4.5 KB alongside SKILL.md
- README.md405 B
- source-index.md1.1 KB
- workflows.md3.0 KB
Gives 0 of the 12 instructions most evals benchmarks skills give in 428 tokens
Counted across 499 of the 513 authors here whose files we hold, read 2026-09-06
- Spawn with-skill and baseline runs in the same turnin 31 of 499, across 24 files
- Keep SKILL.md under 500 linesin 31 of 499, across 24 files
- Draft assertions while test runs are in progressin 31 of 499, across 24 files
- Compare against the baseline after changesin 31 of 499, across 13 files
- Define evals before codingin 26 of 499, across 17 files
- Run evals frequently during developmentin 25 of 499, across 16 files
- Keep evals fastin 24 of 499, across 15 files
- Version evals with codein 24 of 499, across 15 files
- Generate the eval viewer before evaluating outputs yourselfin 24 of 499, across 17 files
- Generate an eval report after runsin 24 of 499, across 15 files
- Track pass@k metrics over timein 22 of 499, across 14 files
- Save a baseline before making changesin 21 of 499, across 9 files
Said here and by no other author read
- Name the chosen workflow and what you are producing
- pick one workflow instead of summarizing
- match the user's goal to a workflow
- structure evals as data, task, and scorers
- feed production logs into datasets and online evals
- unify offline experiments using trace replay
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.