Inference capacity sizing

inference-capacity-sizing

Purpose

Answer "how many GPUs to meet a TPM target" the correct way: start from the customer's interactivity SLA (output tokens/s/user), not the TPM number alone. A TPM target by itself is underdetermined. 3M TPM is 50,000 output tok/s, but that can be a few users each getting many tokens/s or many users each getting few. The tokens/s/user SLA is what picks the concurrency, the concurrency sets the per-pod throughput, and that sets the pod and GPU count. Sizing from a fixed concurrency and back-deriving users is the backwards direction.

When to use

• A customer gives a throughput requirement (TPM, RPS, or total tok/s) and you must size the GPU count.
• You need to sanity-check a sizing recommendation ("N pods for M TPM").
• You want to show a perf optimization as a customer-facing win (delta-GPUs at the SLA).

Do not use this to MEASURE a model (that is inference-perf-bench / inference-aa-workload) or to rank models (that is inference-fleet-leaderboard). This skill consumes an already-measured tok/s/user-vs-concurrency curve.

The method (SLA-first)

SLA S (tok/s/user)  -> c* = max concurrency per replica where tok/s/user(c*) >= S
                    -> per-replica throughput at c*  (= c* * tok/s/user(c*))
                    -> replicas = ceil( (TPM/60) / (per-replica * util) )
                    -> GPUs = replicas * gpus_per_pod

util is the headroom for the TTFT SLA + failover (70 percent is the usual planning number). The pod definition is the GPUs per replica (a GB300 node = 4 GPUs = TP4). Report GPUs as the invariant. A "2x TP4 pod" is 8 GPUs.

Workflow

1. Get the measured curve. Pull the model's tok/s/user anchors from the fleet leaderboards (inference-fleet-leaderboard gives c=1 and c=10. The throughput leaderboard gives the knee), or, for a VERDICT, the full roofline sweep (c=8..128).
2. Run the tool:

python3 ${PROFILE_AND_OPTIMIZE_REPO_ROOT}/tools/capacity_sizing.py \
  --tpm 3000000 --sla "20,50,100,150,215" \
  --anchors "1:215.5,10:122.2,256:41.8" \
  --gpus-per-pod 4 --util 0.70 --model "MiniMax-M2-NVFP4 (GB300 TP4)" \
  --emit <bundle-dir>

It emits the SLA -> c* -> per-pod tok/s -> pods -> GPUs -> $/month table (CAPACITY-SIZING.md + capacity_sizing.json). It fails loud on missing/degenerate inputs (no anchors, non-monotonic curve, tpm<=0). 3. Read the table SLA-first. The pod count is flat for any SLA at or below the throughput knee, then rises steeply as the SLA tightens. Lead with "the GPU count is the SLA you commit to". 4. Match the workload. The curve MUST be measured at the customer's workload shape (ISL/OSL, multi-turn). An AA-shape curve is not valid for a long-context routing-stress ask. Using the wrong curve is the most common sizing error.

Optimization leverage (delta-GPUs)

Translate each validated win into its effect on the GPU count at the SLA-pinned concurrency, each at its own matched baseline + tier (never multiplied together): tuned high-concurrency config (an untuned serve needs several times the GPUs), async scheduling + cudagraph (in the tuned curve, do not disable), load- and KV-cache-aware routing (multi-turn cache-hit, the routing-stress regime), NVFP4-KV capacity (more streams or longer context per pod, MLA models), TP right-size (small-active MoE at TP1, fewer GPUs), MTP (raises tok/s/user so a tight SLA is met at a higher concurrency). Compute the delta-GPUs at the SLA-pinned c from the matched delta for the target workload. Cite the campaign + tier for each.

Full-context reporting (no bare numbers)

Per docs/METHODOLOGY.md "Full-context reporting" (no bare numbers): every number this skill emits MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare tok/s / TPOT / tok/s/user / GPU-count is a defect.

• Identity: model (+HF path), hardware (exact ceiling token GB300/B200), quant, kv-cache dtype.
• Parallelism: TP, DP (replicas), PP, EP.
• Serving cfg: max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode, async_scheduling, prefix-caching.
• Workload: dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts.
• Regime: warm vs cold. Latency vs throughput tier.
• Stack: image/vllm commit, serving engine (vllm/sglang/dynamo).
• Grounding: %SoL + ceiling key, sol_rigor (L1-L4), trials n, same-node, baseline named.
• The sizing inherits the curve's rigor: an anchored + interpolated sizing is a DRAFT. A full-roofline-sweep sizing is a VERDICT. Always state which, and the curve's workload.

Asset validation (review + FAIL LOUD)

Every asset this skill emits (the sizing table / json) is held to docs/METHODOLOGY.md "Asset validation": the tool FAILS LOUDLY on missing/bad/degenerate inputs (no anchors, non-monotonic tok/s/user curve, tpm<=0, util out of range) and never writes a silent placeholder, and the agent REVIEWS the rendered table for human-sense + accuracy (the pod count rises monotonically as the SLA tightens, the knee plateau is present, each number traces to the curve + the formula) and rebuilds it if wrong, never shipping a wrong/confusing sizing with a caveat.

Next lever / BREAKTHROUGH (Grind Mandate)

If this skill emits a measured-curve-backed sizing, its output MUST end by naming the next lever, its expected unlock, and the gate that proves/refutes it, per docs/METHODOLOGY.md "Always be grinding (next-lever framing)". For sizing the standing next lever is: upgrade the DRAFT (anchored + interpolated) to a VERDICT by feeding the full roofline sweep at the customer's workload. Then rank the delta-GPU optimization levers by value x cost and pursue the top one (fewer GPUs at the SLA is the breakthrough). Delete this section ONLY if the skill produces no measurements.

Provenance

Backed by the standalone server/tools/capacity_sizing.py (sibling of tp_rightsize_advisor.py. Pure-Python, fail-loud, unit-tested in test_capacity_sizing.py). Companions: inference-fleet-leaderboard (ranks models. Supplies the tok/s/user anchors), inference-perf-bench (measures the curve).