Performance testing

Performance testing

A benchmark that reports one number from one run is worse than no benchmark: it invites decisions made from noise. Honest measurement accounts for warmup, background variance, and input shape, and it states its uncertainty. Skip the method and you end up optimizing against your cache state and a neighbor's cron job rather than the code.

Method

1. Warm up before you measure. The first iterations pay for JIT compilation, cold caches, and lazy allocation. Run a discard loop until timings stabilize, then start recording. Established harnesses like JMH, pytest-benchmark, and Criterion do this for you; respect it.
2. Fix the input and the input size. Benchmark a known, committed dataset at a stated size, not whatever happened to be in memory. Report the size with the result: "parse, 10k rows" means something, "parse is fast" does not.
3. Run many iterations and report the distribution. Collect hundreds of samples and report median plus a spread such as p95 and standard deviation. The mean alone hides a bimodal result where half the runs hit a slow path.
4. Quiet the machine. Close other load, pin to a core, and disable frequency scaling or turbo boost where you can. On a laptop on battery the numbers are fiction. hyperfine and taskset help isolate a run.
5. Compare against a committed baseline, not a memory. Store last known timings in the repo and diff against them. "20% slower than the baseline we committed" is actionable; "feels slower than last week" is not.
6. Gate regressions in CI with a threshold and repeats. Fail the build when the median exceeds the baseline by a set margin, say 10%, confirmed over repeated runs so a single noisy sample does not block a merge. Track the baseline as it legitimately improves.
7. Profile to find the cost, benchmark to confirm the fix. A profiler like perf, py-spy, or a flame graph tells you where time goes; the benchmark proves the change actually moved the median, not just your intuition.

Signals

• Do two runs of the unchanged benchmark land within your CI threshold of each other? If not, the harness is too noisy to gate on.
• Does the report name the input size, iteration count, and a spread, not just a single millisecond figure?
• When the gate fails, can you point to the committed baseline it regressed against?

Boundaries

This covers micro and function-level benchmarks on one machine. Multi-user capacity, ramp profiles, and saturation belong to load-testing. Whether a given speed is fast enough is a product call against a latency budget, not something a benchmark decides.