Inference optimization agent pack

Inference Optimization Agent Pack

Use this when a team wants an agent to diagnose or improve an AI workload.

This is not a giant memory file. It is a router. Load this file first, then load only one to three sub-skills that match the task.

Core Rule

Inference optimization starts before the model call and ends at energy, capacity, and margin.

Start from the workload shape. Trace down only as far as the evidence demands.

Do Not Load The Whole Pack

Default load:

1. Root rules: AGENTS.md or CLAUDE.md.
2. RTK.md.
3. This router.
4. One to three relevant sub-skills.
5. The templates needed for the output.

Do not load every sub-skill just because it exists. Extra context is not free.

Workflow

1. Measure the task.
2. Diagnose the leaking layer.
3. Change one thing at a time.
4. Verify with task success, p95/p99 latency, cost, and quality.
5. Record the evidence.
6. Decide the next test.

Required Inputs

Ask for or infer these before making recommendations:

• Product surface.
• User task.
• Workload shape: coding, chat/RAG, voice, media, batch, self-hosted, API, or hybrid.
• Current stack: models, providers, engines, tools, sandboxes, caches.
• Traffic: volume, concurrency, request shape, sequence length, media size, call length.
• Success definition.
• Latency target.
• Cost target or pain.
• Trace availability.
• Current failure modes.

If traces are missing, label recommendations as assumptions.

Skill Routing

Use skills/workload-shape-audit/SKILL.md when:

• The workload is unclear.
• The team does not know where the cost or latency goes.
• You need a first WORKLOAD_PROFILE.md.

Use skills/api-spend-recovery/SKILL.md when:

• The team uses APIs, coding agents, RAG, voice APIs, media APIs, or hosted models.
• The likely leaks are model overuse, prompt bloat, RAG bloat, retries, tool loops, cache misses, or rejected outputs.

Use skills/runtime-routing/SKILL.md when:

• The question is API vs self-hosted vs hybrid.
• The team is considering vLLM, SGLang, TensorRT-LLM, Dynamo, Modular MAX, GPUs, CPUs, or edge placement.

Use skills/evidence-review/SKILL.md when:

• Someone claims savings, speedup, quality, lower latency, or better reliability.
• You need to decide whether the evidence is real.

Outputs

Prefer two artifacts:

1. WORKLOAD_PROFILE.md: what the workload is and where it may leak.
2. OPTIMIZATION_RECIPE.md: baseline, diagnosis, change, verification, rollback, next test.

Keep both short enough that an agent can load them later without wasting context.

Hard Rules

• Do not promise automatic savings.
• Do not recommend self-hosting from GPU pricing alone.
• Do not recommend a faster model if retries or failure rate will erase the win.
• Do not call a benchmark useful unless it maps to the product workload.
• Do not treat prompt tokens, GPU utilization, or throughput as the final unit.
• Use cost per successful task, cost per completed call, or cost per accepted asset.
• Separate measured facts, source-backed claims, inferences, and assumptions.
• Preserve product quality. A cheaper failed task is not a win.