Llm system design

LLM System Design

Purpose

Most LLM-system failures are design failures: the model was given a job deterministic code should own, the pipeline shape amplifies cost, or nobody designed what the user sees when the model is wrong. This skill is the design-time discipline — decide whether an LLM belongs at all, give it the narrowest possible role, and design the failure story before the happy path.

Metadata

• Prerequisites: engineering-minimalism (the existence gate below is its ladder applied to LLMs); architecture-analysis when adding to an existing system.
• Related Skills: llm-integration-reliability (implementation-level reliability), agent-engineering (if the design lands on an agent loop), prompt-and-context-engineering (what goes into each call), evaluation-frameworks (proving the design works), system-design (general system design).
• Owns: LLM-role selection (generator / interpreter / judge / router); local-vs-hosted model tradeoffs; pipeline shapes; cost/latency budgeting for LLM calls; design-time measurement/interpretation split.

When to Use / When NOT to Use

Use when:

• Proposing or reviewing any feature that would call an LLM.
• Choosing between a local model and a hosted API, or between model sizes.
• Deciding pipeline shape: one call, a chain, parallel fan-out, or an agent loop.
• An existing LLM feature is too expensive, too slow, or too wrong — redesign starts here, not at the prompt.

Do NOT use (load the sibling instead):

• The design is settled and outputs are misbehaving → llm-integration-reliability.
• You are building the agent loop, tools, or memory → agent-engineering.
• You are writing or tuning the prompt text → prompt-and-context-engineering.
• You need to measure whether the system works → evaluation-frameworks.
• The system has no LLM component → system-design.

Definitions & Mental Model

• Role: the single job the model performs. Four roles cover nearly everything: generator (produce novel text/code), interpreter (turn supplied structured facts into language), judge (score supplied evidence against criteria), router (choose one of N branches).
• Pipeline shape: how calls compose — single call, sequential chain, parallel fan-out over chunks, or agent loop (model chooses next action).
• Measurement/interpretation split: deterministic code computes every number; the model only speaks about numbers it is handed.
• Failure story: the designed behavior when the model is down, slow, or wrong — as much a requirement as the happy path.

A principal engineer treats an LLM as an expensive, non-deterministic, occasionally-wrong subroutine. You would not put such a subroutine anywhere you could avoid it, you would give it the smallest job it can do well, you would bound its blast radius, and you would meter its cost. Every design decision below follows from that framing. The most reliable, cheapest, fastest LLM call is the one your design eliminated.

Core Methodology

1. Run the existence gate. Before any LLM design, ask in order (this is engineering-minimalism's ladder applied to models):

   • Can deterministic code do it? Parsing, arithmetic, lookup, matching, validation, and formatting are NOT LLM jobs — regex, SQL, and a library are cheaper, faster, and always right.
   • Can deterministic code do most of it? Then the LLM's role shrinks to interpreting the deterministic result (see step 3).
   • Is the task actually language-shaped (summarize, explain, judge free text, converse)? Only then does an LLM earn a place.

2. State the job in one sentence with a testable output. "Summarize the diff into ≤3 bullet points" is a design. "Use AI to improve review" is not. If you cannot state the output contract, return to step 1.

3. Assign the narrowest role. Pick exactly one per call:

   • Interpreter is the safest role — the model receives measured facts and produces language about them; it structurally cannot invent the facts.
   • Judge requires ground-truth injection (evidence in the prompt — mechanics owned by llm-integration-reliability step 6).
   • Router requires an enum output contract and a default branch for low-confidence.
   • Generator is the highest-risk role; demand it only when novelty is the product. Apply the measurement/interpretation split: enumerate every quantity in the feature and move each computable one into code.

4. Choose local vs hosted with an explicit tradeoff table. Write the table into the design; do not decide by default:

   Axis	Local (e.g. Ollama)	Hosted API (e.g. Anthropic/OpenAI)
   Data privacy	Data never leaves the machine	Data crosses a trust boundary — check policy
   Marginal cost	~zero per call; hardware fixed cost	Per-token; scales with usage
   Latency	Hardware-bound; no network	Network + queue; usually faster per token on large models
   Capability ceiling	Limited by local hardware	Frontier models available
   Availability	You operate it; fails when host machine does	Provider SLA; fails when network/provider does
   Concurrency	Collapses fast under parallel load — cap it	Rate limits — cap it differently

   Rule: pick the least capable model that passes your evaluation (see evaluation-frameworks); step up only on measured failure, never on vibes.

5. Choose the pipeline shape (decision tree):

What does one unit of work require?
├─ One question, one answer, all context fits comfortably
│    → SINGLE CALL. Default. Cheapest to build, test, and debug.
├─ Output of step A is required input to step B (different roles)
│    → CHAIN. Validate A's output with a deterministic gate
│      before B consumes it (contracts: llm-integration-reliability).
├─ Same operation over many independent pieces (chunks, files, items)
│    → PARALLEL FAN-OUT. Bound chunk size explicitly; cap concurrency
│      with a semaphore sized to the backend; one bad item must not
│      kill the batch (recoverable-per-item taxonomy).
└─ The sequence of steps cannot be known in advance —
   the model must decide what to do next based on observations
     → AGENT LOOP. Last resort: hardest to test, bound, and price.
       Most "agent" ideas are actually chains — re-check.
       If truly needed → agent-engineering.

6. Budget cost and latency at design time. For each call in the shape: estimate tokens in and out, multiply by calls per user action, multiply by expected volume. Write the number down. If a per-action budget does not exist, the design is not finished. Two levers dominate: cache anything generated once and reused (separate generation from evaluation — split owned by llm-integration-reliability step 7), and shrink context (see prompt-and-context-engineering).
7. Design the failure story before the happy path. Answer in the design doc:
   • Backend unreachable → what does the user see? (Fail fast with an actionable message; never a hang.)
   • Output malformed after bounded retries → degraded output, or explicit "unavailable"? (Never silent wrongness.)
   • Output plausible but wrong → what limits the damage? (Human review? Enum constraints? The interpreter role?)
   • Define which failures are fatal vs recoverable per the taxonomy owned by architecture-analysis.
8. Design idempotency for side effects. If the pipeline posts, writes, or publishes anywhere, re-running it must not duplicate output (idempotency owned by architecture-analysis; a worked instance appears in Repository Examples).
9. Record the design as a lightweight ADR (format owned by system-design): role, model choice with tradeoff table, pipeline shape, budgets, failure story, and what you rejected.

Design review checklist

• Existence gate run: every deterministic sub-task moved out of the model
• The model's job stated in one sentence with a testable output contract
• Exactly one role per call (generator / interpreter / judge / router)
• Every computable quantity computed by code, not the model
• Local-vs-hosted decided by written tradeoff table, not default
• Least-capable-model-that-passes-evals selected
• Pipeline shape justified by the decision tree; agent loop only if steps are truly unknowable
• Per-action cost and latency budget written down
• Failure story designed: down / malformed / plausibly-wrong all answered
• Side effects idempotent on re-run
• Evaluation plan exists before build (evaluation-frameworks)

Discovery & Audit Commands

Audit an existing codebase's LLM design before changing it:

# Where are the LLM call sites and which providers?
grep -rn -E "import (anthropic|openai|ollama)|from (anthropic|openai|ollama)|@anthropic-ai|api\.anthropic\.com|api\.openai\.com|localhost:11434" --include="*.py" --include="*.js" --include="*.ts" . | grep -v node_modules

# What roles do the prompts imply? (read each hit)
grep -rln -E "system_prompt|SYSTEM_PROMPT|prompt" --include="*.py" --include="*.js" --include="*.ts" . | grep -v node_modules

# Is there a pipeline shape already? (loops, fan-out, chains)
grep -rn -E "Semaphore|gather|Promise\.all|while|for .* in chunks" --include="*.py" --include="*.ts" . | grep -v node_modules | head -20

# Are budgets/caps designed in? (absence is a finding)
grep -rn -E "max_tokens|MAX_.*CHUNK|budget|timeout" --include="*.py" --include="*.ts" . | grep -v node_modules | head -20

# Which model(s), and is the choice configurable?
grep -rniE "model.*=.*['\"]|OLLAMA_MODEL|claude-|gpt-" --include="*.py" --include="*.ts" --include="*.env.example" . | grep -v node_modules | head -20

Failure Modes & Anti-patterns

Symptom	Mistake	Correction
LLM asked to add numbers, parse dates, or match strings	Skipped the existence gate	Deterministic code for computable tasks; model only for language-shaped work
Model reports metrics that were never measured	Generator role where interpreter belongs	Code measures; model interprets supplied numbers (step 3)
Bill or latency 10× estimate	No per-action budget at design time	Token math × volume, written down, before building (step 6)
Feature hangs when the model host is down	Failure story never designed	Fail fast with actionable message; fatal-vs-recoverable decided up front (step 7)
"Agent" that always executes the same 3 steps	Agent loop chosen for a knowable sequence	It's a chain; agent loops only when the step sequence is genuinely unknowable
Frontier-priced model doing enum classification	Model size chosen by prestige	Least capable model that passes evals; step up on measured failure only
Same content regenerated on every request	Generation fused with per-request work	Cache the generation; split generation from evaluation
Re-runs post duplicate comments/messages	Side effects not idempotent	Delete-or-replace own prior output; design idempotency in (step 8)
Sensitive data sent to a hosted API by default	Local-vs-hosted never actually decided	Write the tradeoff table; privacy axis first
One bad chunk kills a 40-chunk batch	Fan-out without per-item error taxonomy	Recoverable-per-item; log and skip; fatal only for whole-backend failure

Worked Example

Task: "Add AI-powered categorization of customer support tickets into 6 queues."

1. Existence gate: 40% of tickets contain an order-ID pattern that maps deterministically to a queue — regex handles those with zero model calls. Remainder is genuinely language-shaped.
2. Job sentence: "Given ticket title+body, output one of 6 queue names." Output contract: single enum value.
3. Role: router. Enum contract {"queue": "billing|shipping|returns|technical|account|other"}, with other as the low-confidence default routed to a human.
4. Local vs hosted: tickets contain PII → privacy axis dominates → local model, or hosted with a signed data agreement; table written into the ADR.
5. Shape: single call per ticket. Fan-out only exists at the queue-consumer level, which is ordinary code.
6. Budget: ~800 tokens in, ~10 out; 3,000 tickets/day → the token math goes in the ADR; a small local model is plausible — test it first.
7. Failure story: backend down → tickets flow to other (human triage), banner raised for operators; malformed output after 2 strikes → other. Wrongness bounded: worst case equals today's manual triage.
8. Evals before build: 200 historically-labeled tickets as the golden set; the small model ships only if it beats the current keyword rules (evaluation-frameworks).

The resulting design uses the model for 60% of tickets, in the narrowest role, with a failure mode no worse than the status quo.

Repository Examples

Repo facts below are point-in-time illustrations (as of 2026-07-04) — examples, never assumptions about your system.

• prism (~/prism) — a complete design instance: role = judge over injected PR diffs (never recalls code from memory); local model via Ollama (OLLAMA_MODEL, default llama3.2) — privacy + zero marginal cost for a self-hosted reviewer; shape = parallel fan-out over diff chunks bounded by MAX_LINES_PER_CHUNK=120 and asyncio.Semaphore(5); failure story designed: OllamaUnavailableError aborts fast, per-chunk timeout degrades gracefully; scope constrained in the prompt to genuine bugs/security only, with "post nothing" as first-class success; idempotent posting (deletes its own prior comments).
• orphy (~/orphy) — the measurement/interpretation split as architecture: DSP code measures pitch/onsets/cents error; the LLM only interprets those numbers into coaching language — hallucinated measurements are structurally impossible.
• NYTW (~/NYTW) — cost-structure design: question generation is one cacheable call; grading is per-attempt with fresh evidence — generation/evaluation split along cost/frequency lines.
• ragit (~/ragit) — model-selection asymmetry: embedding model fixed (nomic-embed-text — the index's embedding space is a stable contract) while the chat model is a swappable ranked list — the interpreter can change; the index contract cannot.

Validation Criteria

You applied this skill correctly when:

1. The design document states the model's one-sentence job, role, and output contract — and a reviewer can test the contract mechanically.
2. Every quantity in the feature is traceably computed by code or explicitly justified as model-produced.
3. The local-vs-hosted tradeoff table exists with the privacy axis answered.
4. Per-action token/cost/latency budgets are written and arithmetic-checkable.
5. The failure story answers down/malformed/plausibly-wrong, and each failure is classified fatal or recoverable.
6. The pipeline shape matches the decision tree, and any agent loop is justified by genuinely unknowable step sequences.
7. An evaluation plan predates the build.

Provenance & Maintenance

• Sources: ~/prism, ~/orphy, ~/NYTW, ~/ragit — investigated 2026-07-04. Owner context: primary consumers are Sonnet-class agents; LLM output reliability is the owner's hardest recurring problem; minimalism doctrine owner-confirmed 2026-07-04. Methodology is repo-independent; skill authored 2026-07-06.
• Assumptions: the local-vs-hosted table reflects the 2026 provider landscape (pricing models, local-model capability) — re-check axes as the market moves. Repo constants (Semaphore(5), MAX_LINES_PER_CHUNK=120, model names) are point-in-time.
• Re-verification commands:

  grep -rn "Semaphore\|MAX_LINES_PER_CHUNK\|OLLAMA_MODEL" ~/prism/prism ~/prism/.env.example
  grep -rn "nomic-embed-text" ~/ragit/ragit
  

• Likely to drift: provider names and pricing structures; local-model capability ceiling; the example repos' model choices and constants.
• Maintenance checklist:
  • Re-run re-verification commands; re-stamp Repository Examples.
  • Re-validate the tradeoff table axes against the current provider landscape annually.
  • Confirm cross-referenced skills still exist under their directory names.