Brendan

Brendan — systems-performance methodology (the USE method & friends)

Brendan is the measure-don't-guess discipline you run when a system is slow, saturated, or near capacity. It is Brendan Gregg's body of performance methodology — most of all the USE method — distilled into a skill: instead of reaching for whatever tool you remember (the streetlight anti-method) or blaming the component you touched last, you walk a complete list of resources and, for each, check three things — Utilization, Saturation, Errors — until the bottleneck names itself.

USE in one line: For every resource, check utilization, saturation, and errors. It finds ~80% of server issues with ~5% of the effort.

It is the generalization of three metric types applied to every resource:

• Utilization — the average time the resource was busy servicing work, as a percent over an interval. (Caveat: a 5-min average of 20% can hide 100% bursts.)
• Saturation — the degree of extra work the resource can't service, usually queued. Any non-zero saturation is a problem.
• Errors — the count of error events. Check these first: they're quick to read and easy to interpret.

…and the spine that makes it a method: a complete resource list you refuse to leave gaps in — CPUs, memory capacity, network interfaces, storage device I/O, storage capacity, storage/network controllers, and the CPU/memory/I/O interconnects — each crossed off only when U, S, and E are all accounted for.

When to use

Use Brendan whenever you face a resource-bottleneck or capacity question and want a systematic sweep instead of a hunch. Typical shapes:

• "the box/app/cluster is slow" with no obvious cause — run the sweep
• a latency or throughput regression you need to localize to a resource
• capacity planning / "are we near a limit?" / headroom questions
• a saturation symptom (rising queues, tail latency, page-ins, packet drops)
• GPU under-utilization or stalls in an ML/HPC job — note nvidia-smi "GPU-Util" = any kernel running, not SM occupancy, so 100% can still be starved; check true SM activity (dmon sm% / DCGM SM_ACTIVE)
• the first 60 seconds on an unfamiliar Linux host (the triage checklist)
• before you reach for perf/flamegraphs — USE first tells you which resource is worth profiling

When NOT to use

• A pure logic bug with no resource symptom — that's debugging, not perf.
• You already KNOW the bottleneck and just need to fix it — skip the sweep.
• Tuning a cache hit-rate in isolation: caches improve under high utilization, so they don't fit USE; rule out the systemic resources first, then look at cache hit ratios.
• A problem that is fundamentally about latency distribution of one request path rather than a saturated resource — reach for latency analysis (decompose the request's time budget) or off-CPU analysis (an off-CPU flame graph shows what it's waiting on when nothing is saturated); drill-down / flame graphs in references/methodologies.md, not the USE sweep.

The USE procedure (the loop)

Walk the resource list; for each resource, in this order:

1. ERRORS first — easiest to interpret, and a hot resource with errors is worth investigating regardless of load. Non-zero / increasing error counters (including recoverable errors and degraded-but-redundant components) → flag.
2. UTILIZATION — is it busy? 100% is a candidate bottleneck — confirm with saturation before concluding (a device with internal parallelism, e.g. NVMe, can read 100% util yet not be saturated); sustained >70% can already cause queueing delay. Remember averages hide bursts.
3. SATURATION — is work queued/waiting it can't service? Any non-zero value warrants a look (run-queue length, paging, avgqu-sz, drops, retransmits).
4. Interpret & iterate — a resource that's hot and saturated (or erroring) is your suspect. Cross it off only when all three are accounted for; move to the next resource. Don't stop at the first 100% — finish the list so you don't mistake a symptom for the cause.

Before measuring, sketch a functional block diagram of the system (CPUs → memory → I/O → controllers → devices, annotated with max bandwidths): it makes the resource list complete and exposes interconnects people forget. Then apply the same U/S/E lens to software resources (mutex locks, thread/worker pools, process/task capacity, file descriptors) and, in the cloud, to imposed limits (instance/cgroup CPU, memory, net, I/O caps — a tenant can saturate a soft limit while the physical host looks idle).

Full theory (the strategy flowchart, interpretation thresholds, functional diagram, software & cloud resources, caveats): references/use-method.md.

The 60-second triage checklist

On an unfamiliar Linux host, run these ten in order — each peels a different layer (load → errors → CPU → memory → disk → network → processes):

uptime                  # load averages — trend over 1/5/15 min: rising?
dmesg | tail            # recent kernel errors: OOM kills, TCP drops, driver faults
vmstat 1                # r (runq > nCPU = CPU saturation), si/so (swapping), us/sy/id/wa
mpstat -P ALL 1         # per-CPU balance — one hot CPU hides in the average
pidstat 1               # per-process CPU, rolling — who is on-CPU right now
iostat -xz 1            # disks: %util, avgqu-sz, await, r/s+w/s — saturation & latency
free -m                 # memory: available vs used; buff/cache; is free truly low?
sar -n DEV 1            # NIC throughput rxkB/txkB vs known link max → net utilization
sar -n TCP,ETCP 1       # TCP active/passive conn rate, retransmits (saturation/errors)
top                     # overall sanity check: top consumers, state

This is the fast on-ramp to a full USE sweep; the per-resource commands and what each field means are in references/linux-checklist.md.

Mental model: a coroner's checklist, not a hunch

• Resource-complete, not tool-first. Start from the list of resources, then pick the tool that measures each — the opposite of "I know top, let me run top." Missing-tool gaps become known gaps, not silent ones.
• Errors → Utilization → Saturation, per resource. A fixed, boring order so you never skip the cheap signal.
• The bottleneck is the saturated resource, not the busy one. 100% util with zero queue may be fine; 60% util with a growing queue is the problem.
• USE is resource-oriented; TSA is thread-oriented. When the suspect is "my process is slow" rather than "a resource is hot," flip to Thread State Analysis — measure where each thread spends time (Executing, Runnable, Anon-paging, Sleeping/I-O, Lock, Idle) and attack the biggest slice. The two are complementary (references/methodologies.md).

Worked example: "the service got slow after deploy"

1. 60-second triage. uptime shows load 30 on 8 CPUs (high). vmstat 1: r=24 (run-queue ≫ 8 → CPU saturation), si/so=0 (not swapping), wa low (not disk-bound). dmesg | tail is clean (no errors).
2. USE sweep, CPU resource. Errors: perf/MCE clean. Utilization: mpstat -P ALL 1 → all 8 CPUs ~95% in %usr (not %sys, not %iowait) — genuinely compute-bound, evenly balanced. Saturation: vmstat r confirms a deep run-queue; sar -q runq-sz agrees. CPU is the bottleneck (hot and saturated, in user time).
3. Cross off the rest so you don't fix a symptom. Memory free -m: ample available, vmstat paging zero — not memory. Disk iostat -xz 1: %util low, avgqu-sz<1 — not disk. Net sar -n DEV/TCP: throughput modest, retransmits ~0 — not network. List is clean except CPU-user.
4. Now—and only now—profile the hot resource. A CPU flame graph (perf record -F 99 -ag -- sleep 30 → flamegraph, or profile -F 99) shows the new deploy's JSON serializer dominating on-CPU time. That's the cause; USE pointed the profiler at the right resource instead of guessing.

VERIFY BEFORE CLAIM: report the actual numbers (r=24/8 CPUs, %usr=95, the flame-graph frame %), not "seems CPU-bound."

Quick start

1. Triage in 60 seconds to localize the layer:

   bash skills/brendan/templates/60s-triage.sh     # runs the ten checks, annotated
   

2. Open a USE worksheet and walk the resource list — one row per resource, filling U / S / E so gaps are visible:

   cp skills/brendan/templates/use-worksheet.md use-worksheet.md
   # for each resource: record the U/S/E metric, its value, and a verdict
   

3. Look up the exact command for any (resource × U/S/E) cell in the Linux checklist; for a GPU job use the GPU checklist (nvidia-smi, DCGM, XID/NVLink):

   # references/linux-checklist.md  — CPU/mem/net/disk/controllers/interconnects + software resources
   # references/gpu-checklist.md    — nvidia-smi dmon/pmon, DCGM dcgmi/dcgm-exporter, XID/ECC/NVLink, Nsight; ROCm equivalents
   

4. The bottleneck = the resource that's hot and saturated (or erroring). Only then profile it (flamegraph / drill-down / off-CPU — see references/methodologies.md). Feed the one scalar it gives you (p99, %util, queue depth) into an andrej loop to drive the fix and prove it stuck.

References

• references/use-method.md — the USE method in depth: the three metric types and how to interpret each, the complete resource list, the strategy flowchart, the functional block diagram, software resources, cloud/virtualization limits, and the caches caveat.
• references/linux-checklist.md — the full USE Method Linux checklist: the exact command(s) for every (resource × utilization/saturation/errors) cell — CPU, memory, network, storage I/O & capacity, controllers, interconnects, plus software resources (kernel/user mutex, task capacity, file descriptors) — and the 60-second triage checklist and the observability-tool map.
• references/gpu-checklist.md — USE applied to GPUs: NVIDIA via nvidia-smi, nvidia-smi dmon/pmon, --query-gpu, DCGM (dcgmi dmon, dcgm-exporter, field IDs), XID/ECC/retired-pages errors, NVLink/PCIe counters & topology, throttle reasons, and profilers (Nsight Systems/Compute); AMD ROCm equivalents.
• references/methodologies.md — the rest of the toolkit and what to use instead of guessing: TSA (thread-state), workload characterization, drill-down analysis, latency analysis, off-CPU analysis, CPU/off-CPU flame graphs, the RED method for services, and the named anti-methods to avoid (streetlight, drunk-man, blame-someone-else, random-change).

Templates

• templates/use-worksheet.md — a fill-in worksheet: the resource list with a U/S/E cell per resource, a verdict column, and a functional-block-diagram slot, so an investigation is complete and auditable.
• templates/60s-triage.sh — a runnable, annotated 60-second triage script (the ten commands above) that prints what to look for in each.

External sources

• Brendan Gregg, The USE Method — brendangregg.com/usemethod.html; the Linux checklist — brendangregg.com/USEmethod/use-linux.html.
• Brendan Gregg, TSA Method — brendangregg.com/tsamethod.html; Flame Graphs — brendangregg.com/flamegraphs.html; off-CPU analysis, bcc/bpftrace tools, and the book Systems Performance (2nd ed.).
• The 60-second checklist (Netflix TechBlog, "Linux Performance Analysis in 60,000 Milliseconds").

Provenance. This skill was distilled from the four pages above (USE method, USE Linux checklist, TSA, 60-second) plus general systems-performance knowledge — it is not a page-by-page crawl of brendangregg.com. The GPU / nvidia-smi / DCGM / NVLink, flame-graph, and bcc/bpftrace tooling content is knowledge-derived; verify exact flags and field IDs against current vendor/tool docs before relying on them.

Relationship to andrej & terry

Brendan tells you WHICH resource to attack and WHAT to measure; andrej is the loop that drives the fix from that measurement and proves it stuck. They compose directly:

• A USE sweep ends with one scalar (p99 latency, %util, queue depth, val throughput). That scalar is exactly the metric an Andrej iteration gates keep-or-revert on — Brendan picks the metric, Andrej moves it.
• Brendan's structured signals (a --query-gpu CSV stream, sar/iostat counters, DCGM exporter) are the structured polling Andrej prefers over log-grep, and Brendan's "report the real numbers" is Andrej's verify-before- claim.
• Under terry, Brendan is what a worker runs to localize a perf bottleneck in its scoped task before iterating; the fleet then runs many such loops in parallel. Use Brendan to find the bottleneck, Andrej to fix one, and terry to fix many at once.