Performance profiler

Performance Profiler

Overview

Make slow things fast — correctly and provably. This skill enforces a measure-driven loop: never optimize on a hunch, always profile to find the real hot path, fix the biggest contributor first, then re-measure to confirm the win and guard against regressions.

Keywords: performance, profiling, optimization, bottleneck, latency, throughput, p99, slow, benchmark, flamegraph, CPU profile, memory leak, allocations, N+1 query, caching, big-O, complexity, hot path, regression.

The cardinal rule: measure first. Most "obvious" optimizations target the wrong code. Profilers routinely show that 90% of time sits in a place nobody suspected.

Workflow

Follow this loop. Do not skip steps — especially step 1 and step 6.

1. Define the goal and a metric. Pick ONE primary metric and a target: wall-clock latency (p50/p95/p99), throughput (req/s, rows/s), CPU time, peak memory (RSS), or allocations. Write down the current value and the target. "Make it faster" is not a goal; "cut p95 from 800ms to under 200ms" is.

2. Reproduce reliably. Build a repeatable scenario with representative data volume. A bottleneck at 10 rows may vanish at 10M and vice-versa. Disable noise: warm caches, JIT warmup, fixed input, quiet machine, multiple runs.

3. Measure the baseline. Time/benchmark the whole operation before touching anything. Save the numbers. Use scripts/bench.py for a quick statistically sane wall-clock benchmark of a Python callable or shell command.

4. Profile to find the hot path. Use a real profiler (not scattered print timers) to attribute cost. Find the function/line/query consuming the most time or memory. See references/profiling-tools.md for the right tool per language and how to read its output.

5. Diagnose and fix the top contributor. Apply the cheapest effective fix from the optimization hierarchy (see below). Change ONE thing at a time so each change's impact is attributable.

6. Re-measure and verify. Re-run the exact baseline benchmark. Confirm the metric improved and correctness is unchanged. Quantify: "p95 800ms → 180ms, -77%". If no improvement, revert and re-profile.

7. Repeat or stop. Return to step 4 for the next hot path, or stop when the target is met. Add a regression guard (a benchmark assertion or CI check) so the win doesn't rot.

Optimization hierarchy

Apply fixes in this order — cheapest/highest-leverage first. Most wins come from the top three.

1. Do less work. Remove redundant computation, dead code, needless copies, logging in hot loops. Cache or memoize pure, repeated results. Hoist invariants out of loops. Compute lazily / short-circuit.
2. Reduce algorithmic complexity. Replace O(n²) with O(n log n) or O(n): use a hash set/map for membership and lookups, sort once instead of repeatedly scanning, use the right data structure. See references/complexity-cheatsheet.md.
3. Fix I/O and data access. Batch round-trips, eliminate N+1 queries, add the right index, select only needed columns, stream instead of buffering, use connection pooling, paginate. I/O usually dwarfs CPU.
4. Parallelize / concurrency. Overlap independent I/O (async), use a worker pool for CPU-bound work, vectorize (NumPy/SIMD). Only after single-thread work is minimized — parallelizing a bad algorithm just burns more cores.
5. Reduce allocations / memory pressure. Reuse buffers, avoid intermediate collections, use generators/iterators, pick compact representations, cut GC churn.
6. Lower-level / micro-optimizations. Compiled extensions, better serializer, tuned runtime flags, JIT-friendly code. Last resort — small payoff, high maintenance cost.

Decision framework

• CPU-bound vs I/O-bound first. If wall time >> CPU time, you are waiting on I/O (disk, network, DB, locks) — chase step 3/4, not micro-CPU tuning. If wall time ≈ CPU time, attack the algorithm (step 1/2).
• Latency vs throughput. Caching and batching help throughput; they may not help a single cold request's latency. Optimize for the metric you committed to.
• Tail vs average. p99 spikes are usually GC pauses, lock contention, cold caches, or a slow dependency — not the average path. Profile the slow requests specifically.
• Amdahl's law. Speeding a section that's 5% of runtime can yield at most a 5% gain. Always spend effort proportional to a section's share of total cost.

Best Practices

• Profile in a configuration that resembles production (data size, build flags, release/optimized mode — never profile a debug build and extrapolate).
• Change one variable per measurement; keep a running log of (change → metric).
• Run benchmarks multiple times; report median and spread, not a single run.
• Keep correctness tests green throughout — a fast wrong answer is worthless.
• Commit the baseline numbers and the proof of improvement alongside the change.
• Prefer eliminating work over doing the same work faster.
• Add a regression benchmark to CI for anything you fought hard to speed up.

Common Pitfalls

• Optimizing without profiling — fixing code that isn't the bottleneck.
• Premature optimization — complicating code for gains the user never feels.
• Micro-benchmark lies — measuring a case the optimizer elides, or one with unrepresentative data/caching, then shipping a non-win.
• Benchmarking debug builds or with profiler overhead included.
• Single-run conclusions — noise mistaken for signal.
• Premature parallelism — adding threads/async over an O(n²) core.
• Cache without invalidation strategy — turns a speed bug into a correctness bug.
• Ignoring the tail — celebrating a better mean while p99 still times out.

Supporting files

• references/profiling-tools.md — per-language profiler commands (Python, Node/JS, Go, Java, SQL, web/browser), what each measures, and how to read flamegraphs and call trees.
• references/complexity-cheatsheet.md — big-O of common operations and data structures, plus the canonical "swap this for that" optimization patterns (N+1 fix, set-membership, memoization, batching).
• scripts/bench.py — runnable benchmark harness: times a Python expression or a shell command over N runs and reports min/median/mean/stdev with a clean comparison mode for before/after.
• examples/optimize-n-plus-one.md — full worked example taking a slow endpoint from 1.9s to 60ms through the whole loop (measure → profile → fix → verify).