Performance engineering

Performance Engineering

Purpose

Almost all performance work fails the same way: optimizing before measuring, then declaring victory without re-measuring. This skill enforces the iron law — define the metric, reproduce the workload, profile to find where time actually goes, fix the biggest measured contributor via the optimization ladder, and verify with the same measurement — so every claimed speedup is a number, not a feeling.

Metadata

• Prerequisites: debugging-playbook's one-variable discipline (shared law); proof-and-analysis for complexity/estimation mechanics.
• Related Skills: observability-and-diagnostics (production measurements come from its instrumentation), validation-and-testing (regression checks), engineering-minimalism (when NOT to optimize), llm-system-design (LLM-call cost budgets).
• Owns: profiling; measurement-before-optimization; the optimization ladder; performance regression guarding; benchmark discipline.

When to Use / When NOT to Use

Use when:

• Anything is reported slow, expensive, or memory-hungry — before touching code.
• Asked to optimize, benchmark, or set performance targets.
• Reviewing a diff whose justification is "faster" (demand the numbers).
• Choosing between implementations on performance grounds.

Do NOT use (load the sibling instead):

• Output is WRONG, not slow → debugging-playbook.
• The cost is LLM tokens at design time → llm-system-design step 6 (return here for measuring a live pipeline).
• No one has reported or measured a problem, and no target exists → do not optimize; load engineering-minimalism and move on (premature optimization is speculative code).
• The system lacks the instrumentation to measure production behavior → observability-and-diagnostics first.

Definitions & Mental Model

• Metric: the number being improved — latency (p50/p95/p99, not averages), throughput (units/sec), memory (peak RSS), or cost (per request/day). One primary metric per effort.
• Target: the value at which you STOP. Without a target, optimization never ends and never succeeds.
• Profile: a measurement of where time/memory actually goes, attributed to code locations.
• Amdahl reasoning: overall speedup is capped by the fraction you optimize — a 100× improvement of something taking 5% of the time yields at most ~5%. Always optimize the biggest measured contributor first.
• Benchmark: a repeatable measurement under a controlled workload; only comparable when workload, environment, and method are held constant.

Mental model: performance work is debugging where the bug is a number. The same laws apply: reproduce first (a workload, not a vibe), let evidence pick the suspect (the profile, not intuition — engineers' guesses about hot spots are notoriously wrong), change one variable, and prove the fix by re-running the original measurement. The only new law is Amdahl's: the profile is not just a suspect list, it is a budget — the top entry bounds what any fix elsewhere can achieve.

Core Methodology

1. Define the metric and the target before anything else. "The export endpoint's p95 must drop from 8s to under 2s at current data volume." No metric+target → no optimization work; get the target first (from requirements, or set a defensible one and state it).
2. Reproduce the workload. A measurable, repeatable scenario: fixed input data, fixed environment, recorded invocation. Percentiles need many runs; single-shot timings are noise. If the slowness only exists in production, first get measurements from there (observability-and-diagnostics), then build the smallest local workload that shows the same shape.
3. Measure the baseline and write it down. Number, date, environment, method. This is the "before" that every claim will be judged against.
4. Profile — never guess. Standard tools by ecosystem (mark: install-required where noted):
   • Python: python -m cProfile -s cumtime script.py (built-in); py-spy top --pid <PID> for live processes (install required).
   • Node: node --prof app.js then node --prof-process isolate-*.log (built-in); clinic suite (install required).
   • SQL: EXPLAIN / EXPLAIN ANALYZE on the slow query — the database will tell you about missing indexes and sequential scans.
   • Whole commands: time <cmd> (built-in); hyperfine '<cmd>' for statistical comparison of two commands (install required).
   • Memory (Python): tracemalloc (built-in). Read the profile top-down: the top few entries are the entire opportunity (Amdahl). If the top entry is I/O wait, no CPU optimization will help — the ladder's rungs change accordingly.
5. Fix via the optimization ladder — take the highest rung that applies to the biggest contributor:

What does the profile blame?
├─ RUNG 1 — Algorithm/complexity: an O(n²) loop, N+1 queries,
│    repeated recomputation of an invariant, missing index.
│    → Fix the complexity class. Routinely 10-1000×. Always first.
├─ RUNG 2 — Batching & caching: per-item calls to something remote
│    or expensive (DB, API, embedding service, disk).
│    → Batch the calls (one round-trip for N items); cache what is
│      recomputed with an explicit invalidation story. Often 10-100×.
├─ RUNG 3 — Concurrency: independent I/O-bound work done serially.
│    → Parallelize WITH a cap (semaphore/pool sized to the backend);
│      concurrency without caps trades slowness for outages.
│      CPU-bound work needs processes/threads per runtime rules.
└─ RUNG 4 — Micro-optimization: hot inner loop, allocation churn,
     serialization overhead. → Last resort, only with a profile
     entry to point at, and only after rungs 1-3 are exhausted.
     Usually <2× and pays in readability.

6. Change one variable, then re-measure with the SAME method as step 3. Same workload, same environment, same command. Improvement is before → after on that measurement — nothing else counts. If the gain is within run-to-run noise, report "no detectable improvement" and revert the complexity you added.
7. Stop at the target. Meeting the target ends the effort — further optimization is unrequested code (engineering-minimalism). Record the final numbers next to the change (commit message or PR description) so the claim is auditable.
8. Guard against regression. For the paths that mattered enough to optimize: keep the benchmark script in the repo, record its expected range, and re-run it when that code changes. A number in a script beats a memory of one. (For a check that must run in the test suite, a generous threshold assertion beats a flaky exact one — mechanics: validation-and-testing.)

Optimization checklist

• Metric (with percentile) and stop-target written down first
• Workload reproducible: fixed data, environment, invocation, enough runs for stable numbers
• Baseline recorded: number, date, method
• Profile taken; the fix targets the TOP measured contributor
• Ladder rung justified (complexity before batching before concurrency before micro)
• One variable changed; re-measured with the identical method
• Noise-level gains reported honestly and reverted
• Stopped at target; final numbers recorded with the change
• Benchmark script + expected range kept for regression re-runs

Discovery & Audit Commands

# --- Baseline & comparison ---
time python script.py                      # coarse built-in timing (or: time node app.js)
hyperfine 'python script.py'               # statistical benchmarking (install required)

# --- Profiling ---
python -m cProfile -s cumtime script.py 2>&1 | head -30     # Python, built-in
py-spy top --pid <PID>                                       # live Python (install required)
node --prof app.js && node --prof-process isolate-*.log | head -40   # Node, built-in
# SQL: run inside the DB shell:  EXPLAIN ANALYZE <the slow query>;

# --- Memory ---
/usr/bin/time -l python script.py 2>&1 | grep -i "maximum resident"   # macOS peak RSS (-v on Linux)

# --- Find the classic complexity smells before profiling confirms them ---
grep -rn "for .* in .*:" --include="*.py" . | grep -v node_modules | head    # then eyeball for nested loops over the same data
grep -rn -E "\.get\(|SELECT" --include="*.py" . | grep -iE "for |while " | head   # queries inside loops (N+1 candidates)

# --- Are there existing benchmarks/targets in this repo? ---
find . -iname "*bench*" -not -path "*/node_modules/*" -not -path "*/.git/*" | head
grep -rn -iE "p95|p99|latency|timeout" --include="*.md" --include="*.py" . | grep -v node_modules | head

Failure Modes & Anti-patterns

Symptom	Mistake	Correction
Week of optimization, users notice nothing	Optimized a 5% contributor	Profile first; Amdahl budgets the effort (steps 4-5)
"It feels faster" ships	No baseline, no re-measurement	Same-method before/after numbers or it didn't happen (steps 3, 6)
Micro-optimized inner loop, O(n²) untouched	Ladder inverted	Complexity class first, always (step 5)
Parallelized, now the backend falls over	Concurrency without caps	Semaphore/pool sized to the target; rung 3 includes the cap (step 5)
Cache added, stale data bugs follow	Caching without an invalidation story	Cache = value + lifetime + invalidation, designed together (step 5)
Average latency fine, users furious	Averages hide the tail	Percentiles (p95/p99) as the metric, never the mean (step 1)
Optimization never ends	No stop-target	Target defined first; meeting it ends the work (steps 1, 7)
Regression sneaks in a month later	Benchmark existed only in a terminal scrollback	Script + expected range live in the repo (step 8)
Two "benchmarks" compared across laptops	Method not held constant	Same workload, environment, command — or no comparison (step 6)
Unreadable code for a 3% gain	Rung-4 work without exhausting rungs 1-3	Micro-optimization last, profile-pointed, gain-justified (step 5)

Worked Example

Task: "the nightly sync takes 40 minutes; it must finish under 10."

1. Metric/target: wall-clock of the sync job; target <10 min at current volume (12k records).
2. Workload: a copy of one night's input, run via time python sync.py --input fixture/. Three runs: 39–41 min. Baseline: 40 min.
3. Profile: python -m cProfile -s cumtime sync.py ... → 91% of cumulative time in fetch_details(), called 12,000 times — one HTTP request per record. The tempting suspect (a gnarly transform function someone wanted to rewrite) is 2%. Amdahl says the rewrite was worth at most ~1 minute.
4. Ladder: rung 2 — batching. The API offers a bulk endpoint (100 records/request). Change ONE thing: fetch_details → fetch_details_batch, 120 requests instead of 12,000.
5. Re-measure, same method: three runs: 6.5–7 min. 40 → ~7 min. Target met.
6. Stop. The 2% transform stays un-rewritten. A bench/sync_bench.sh with the fixture and the expected range (≤10 min) lands next to the code; the PR description carries the before/after numbers.

Repository Examples

Limited repository evidence available (as of 2026-07-04): none of the investigated repositories contains profilers, benchmarks, or recorded performance targets — performance tooling is an evidence gap in the source corpus, which is why the worked example above is generic.

The closest real artifacts:

• ragit (~/ragit) — rung 2 in production code: indexer.py batches embedding calls 64 at a time instead of per-chunk requests to the embedding service — batching-beats-per-item for remote calls, the exact shape of the worked example's fix.
• prism (~/prism) — rung 3 with the cap included: asyncio.Semaphore(5) parallelizes chunk reviews while bounding concurrency to what a local Ollama backend survives; MAX_LINES_PER_CHUNK=120 bounds per-unit work — concurrency and caps designed together.

Validation Exercise (any repository): pick the slowest command/endpoint you know of; write down a metric and target; capture a 3-run baseline with time; profile it; name the top contributor and which ladder rung applies. Stop there — the exercise is the measurement discipline, not the fix.

Validation Criteria

You applied this skill correctly when:

1. A written metric + target predates any code change.
2. The baseline and final numbers exist, produced by the identical method, and the delta is above noise.
3. The fix targets the profile's top contributor, and you can state the ladder rung and why higher rungs didn't apply.
4. Exactly one variable changed per measured comparison.
5. Work stopped at the target, and no rung-4 cleverness shipped without a profile entry justifying it.
6. A regression benchmark (script + expected range) lives in the repo for the optimized path.

Provenance & Maintenance

• Sources: ~/ragit, ~/prism — investigated 2026-07-04; the profiling tool list reflects standard 2026 tooling. Skill authored 2026-07-06; methodology is repo-independent.
• Assumptions: tool availability varies (py-spy, hyperfine, clinic are install-required and marked as such); /usr/bin/time -l is the macOS flag (-v on GNU/Linux) — verified for the authoring platform, adjust per OS.
• Re-verification commands:

  grep -n "batch" ~/ragit/ragit/indexer.py | head
  grep -rn "Semaphore" ~/prism/prism
  python -m cProfile --help >/dev/null && echo "cProfile OK"
  

• Likely to drift: profiler tooling per ecosystem; the example repos' batch sizes and caps.
• Maintenance checklist:
  • Re-run re-verification; re-stamp Repository Examples.
  • Refresh the tool list against current ecosystem standards annually.
  • If an owner repo gains real benchmarks, promote them into the Repository Examples.