agentsclimarketplace

Benchmark design

Skill Amey-Thakur/AI-SKILLS/skills/performance/benchmark-design

Plug-and-play skills and prompts for every AI coding agent

Install
npx -y skills add Amey-Thakur/AI-SKILLS --skill benchmark-design

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 19 days oldThe repository was created 19 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
  • 4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Write benchmarks that do not lie, controlling warmup, isolation, and statistics so the number reflects the code and not the environment. Use when comparing implementations, defending a performance claim, or a microbenchmark result looks too good.

SKILL.md

3.0 KB, as published. Nobody here has run it

Benchmark design

A benchmark is an experiment, and most of them are broken experiments that measure the JIT warming up, the allocator settling, or the dead-code eliminator deleting the thing under test. A confident wrong number is worse than no number, because someone ships a decision on it. Design the measurement so the result survives being run again on another machine.

Method

  1. State the question and the unit first. "Nanoseconds per parse() call on 1 KB inputs", not "is it fast". Fix the input size, the data shape, and what counts as one operation before writing any timing code.
  2. Warm up until steady state, then measure. Run the workload untimed until the JIT compiles and caches fill (JMH @Warmup, Criterion and pytest-benchmark do this automatically). Discard warmup samples. A cold first run measures compilation, not the code.
  3. Defeat dead-code elimination. Consume every result: return it, feed it to a blackhole (Blackhole.consume in JMH), or accumulate into a checksum you print. A loop whose output is unused legally compiles to nothing, and then you are timing an empty loop.
  4. Isolate the machine. Pin to a core (taskset -c 2), disable turbo and frequency scaling (cpupower frequency-set -g performance), close background load, and run on hardware, not a shared CI runner or a laptop on battery. Report the CPU, OS, and runtime version alongside the number.
  5. Report a distribution, not a mean. Collect many samples and show median plus p95 and the interquartile range. The mean hides multimodal results from GC pauses and context switches; the median with spread tells the truth. Flag any run whose variance exceeds a few percent as untrusted.
  6. Compare with a real significance test. For A versus B, run interleaved samples and apply a Mann-Whitney U test or non-overlapping confidence intervals, not eyeballed averages. A 3% difference inside the noise band is not a difference.

Litmus tests

  • Does re-running the benchmark on a second machine preserve the ranking of the variants, even if absolute numbers shift?
  • If you delete the code under test, does the benchmark get faster (it should break or error, proving the work was actually happening)?
  • Do you report median and spread with hardware and runtime, not a lone mean?
  • Would the measured difference survive a significance test on fresh samples?

Boundaries

This governs microbenchmarks and small controlled comparisons. Whole-system load behavior under concurrency belongs to throughput-scaling and latency-analysis; catching drift over time in automation is performance-regression-detection. Production numbers come from real-user-monitoring, not a bench.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.