Performance profiling

Performance Profiling

A structured approach to diagnosing and fixing performance problems. The workflow is the same across languages; the tools differ.

Rule zero: Profile before optimising. Optimising without data is guessing. Guessing wastes time and often makes things worse.

---

Workflow

1. Define the problem

Write one sentence: "The GET /users endpoint takes 4s at p95 under 50 concurrent users; the target is < 500ms."

Without a measurable baseline and a concrete target, you won't know if your optimisation worked.

2. Form a hypothesis

Based on the symptoms, guess the likely cause:

• Slow endpoint → database query (N+1, missing index, large result set)?
• High CPU → tight loop, regex, serialisation?
• High memory → unbounded cache, large object held in scope, leak?
• Slow startup → heavy imports, unnecessary initialisation?

3. Profile

Pick the appropriate tool (see below) and run it against your hypothesis. Look at the top 5–10 most expensive functions/frames.

4. Benchmark

Measure before you change anything. Write a reproducible benchmark:

# HTTP endpoint (Apache Bench)
ab -n 1000 -c 50 http://localhost:3000/users

# Or hey (Go-based, better output)
hey -n 1000 -c 50 http://localhost:3000/users

Record p50, p95, p99 and throughput. This is your baseline.

5. Fix and re-benchmark

Make one change at a time. Re-run the benchmark and compare to the baseline. Multiple simultaneous changes make it impossible to know which one helped.

6. Verify under realistic load

Profiling under idle or synthetic load can miss real bottlenecks. Use production-shaped data volumes and concurrency levels.

---

Python

cProfile (deterministic, low overhead)

import cProfile
cProfile.run('my_function()', sort='cumulative')

Or on the command line:

python -m cProfile -s cumulative my_script.py | head -30

Key columns: tottime (time in this function only) and cumtime (including all called functions). Focus on functions with high tottime or unexpectedly high call counts.

py-spy (sampling, attach to live process)

pip install py-spy

# Flame graph for a running process
py-spy record -o profile.svg --pid <PID>

# Live top view
py-spy top --pid <PID>

Use py-spy when you can't modify the code or need to profile a production-equivalent workload without restarting.

Memory — tracemalloc

import tracemalloc
tracemalloc.start()

# ... code to profile ...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
    print(stat)

---

Go

pprof (built-in)

Add the pprof HTTP handler to your server:

import _ "net/http/pprof"
// (pprof registers its handlers on the default mux)
go http.ListenAndServe("localhost:6060", nil)

Then collect and analyse:

# 30-second CPU profile
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Heap allocation snapshot
go tool pprof http://localhost:6060/debug/pprof/heap

# Interactive web UI
go tool pprof -http=:8080 cpu.prof

In the pprof UI, look at the flame graph. The widest frames are where the most time is spent.

Benchmarks (built-in)

func BenchmarkFindUsers(b *testing.B) {
    for i := 0; i < b.N; i++ {
        FindUsers(ctx, filter)
    }
}

go test -bench=BenchmarkFindUsers -benchmem -count=5 ./...

-benchmem shows allocations per operation; high alloc counts often point to unnecessary object creation in hot paths.

---

Node.js / TypeScript

Built-in V8 profiler

node --prof app.js
# Produces isolate-*.log

# Process the log
node --prof-process isolate-*.log > profile.txt

clinic.js (higher-level, recommended)

npm install -g clinic

# CPU flame graph
clinic flame -- node app.js

# Event loop and async I/O analysis
clinic bubbleprof -- node app.js

# Doctor (recommends which tool to use)
clinic doctor -- node app.js

Chrome DevTools

Run with --inspect and open chrome://inspect:

node --inspect app.js

Use the Performance tab for flame charts; Memory tab for heap snapshots and allocation timelines.

---

Reading a flame graph

A flame graph shows call stacks. Width represents time (wider = more time spent).

• Look for wide frames near the top — those are the hot functions.
• Wide frames near the bottom are entry points (expected).
• Flat tops (no children take significant time) mean the function itself is the bottleneck, not something it calls.
• Unexpectedly wide frames from serialisation libraries, ORMs, or string processing are usually quick wins.

---

Common findings and fixes

Finding	Likely cause	Fix
DB query in hot loop	N+1	Eager load / batch query
Large allocations	Unnecessary copies, missing reuse	Pool / buffer reuse
Serialisation dominates	JSON.stringify / pickle in hot path	Cache, reduce frequency
GC pressure (Go)	Many small allocations	Pre-allocate slices, sync.Pool
Event loop blocking (Node)	Sync I/O or CPU in hot path	setImmediate, worker thread
Import time (Python)	Heavy top-level imports	Lazy imports, startup profiling

---

Review checklist (for performance-sensitive PRs)

• Baseline benchmark recorded before the change
• Post-change benchmark shows measurable improvement
• No N+1 queries introduced in data access layer
• No synchronous I/O in Node.js request handlers
• No unbounded caches (LRU or TTL-bounded)
• Database indexes exist for all new query filters