Cost guardrail

Cost Guardrail

The most expensive model is the one running on every request when it does not need to.

LLM cost is not a finance problem — it is an architecture problem. The design determines the bill. This skill enforces cost-awareness as a first-class design constraint, not an afterthought.

When to use

• Designing any system that calls an LLM (directly or via an agent)
• Before scaling a workload to higher volumes
• When a cost estimate is required for a feature or release
• When reviewing an architecture for unbounded cost vectors
• When choosing between model tiers for a given task

Procedure

1. Identify every LLM call in the system — list: which agent or component makes the call, the model tier used, the approximate input and output token counts, and the call frequency (per user action / per minute / per batch).

2. Apply the model tiering test — for each LLM call, ask:

   • Does this task require deep reasoning, or is it classification / extraction / reformatting?
   • Can the task be completed with a smaller or faster model?
   • Is the model tier choice based on evidence (benchmark, A/B test) or assumption?

   General tiering principle (verify current pricing against your provider's documentation before relying on it):

   Task type	Appropriate tier
   Simple classification, extraction, summarisation	Small / fast model
   Complex reasoning, multi-step planning, code generation	Mid-tier model
   Deep analysis, architecture decisions, adversarial review	Highest-tier model

3. Identify unbounded cost vectors — flag any call pattern where the token count or call volume has no upper bound:

   • Loops that call an LLM until a condition is met (with no max-iteration guard)
   • User-triggered calls with no rate limiting
   • Context windows that grow unboundedly across a conversation
   • Batch jobs with no per-run budget ceiling

4. Estimate the monthly cost envelope — for each LLM call:

   estimated monthly cost ≈ (input tokens × input price) + (output tokens × output price) × calls/month
   

   Use current published rates from your provider. Do not use rates from training data — they change.

5. Add cost controls — for each unbounded vector:

   • Set a max-token budget per call (trim context if needed)
   • Add rate limiting at the application layer
   • Add a budget alert at the infrastructure layer
   • Consider caching repeated calls with identical or near-identical inputs

6. Check for caching opportunities — LLM calls that return the same result for the same input are cacheable. Prompt caching (where supported by the provider) can reduce cost significantly on repeated prefixes.

7. Document the cost model — in the ADR or design doc, record: model tiers chosen, rationale, estimated monthly cost at target scale, and the controls in place.

Outputs

• LLM call inventory: component | model tier | input tokens (est.) | output tokens (est.) | frequency | monthly cost (est.)
• Unbounded cost vectors flagged with mitigations
• Monthly cost estimate at target scale
• Recommended model tier per call with rationale

Guardrails

• Never use pricing from training data. Rates change. Fetch current rates from the provider's documentation before estimating.
• A call that "works" at low volume may be unaffordable at scale. Always estimate at the target scale, not the current scale.
• Caching is not optional for high-frequency repeated calls. An uncached LLM call repeated thousands of times per day is a design flaw.
• Token budgets are architecture decisions. Decide them explicitly; do not let the model decide by consuming whatever context is available.

Anti-rationalization table

Excuse	Counter
"It's only a few cents per call"	At scale, cents become thousands of dollars. Estimate the monthly envelope.
"We'll optimise later"	Cost optimisation is hardest after the architecture is set. Do it now.
"The big model gives better results"	Verify with a test. Small models are often sufficient for structured tasks.
"We don't know the volume yet"	Estimate a range. A 10x cost swing between low and high volume is a design risk.