Llm integration reliability

LLM Integration Reliability

Purpose

LLMs are non-deterministic text generators wired into deterministic programs; every integration point is a trust boundary. This skill is the runbook for making any codebase survive malformed JSON, hallucinated facts, endless retry loops, and flaky backends. The core stance: never trust model output — contract it, extract it defensively, validate it, cap it, and fail loudly with the raw text attached when it still breaks.

When to Use / When NOT to Use

Use when:

• Writing or reviewing any code that sends a prompt and consumes the response programmatically (JSON, enums, scores, tool calls).
• Building agent loops (ReAct, tool-calling, multi-step) on any backend, local or cloud.
• Building LLM-as-grader / LLM-as-reviewer / LLM-as-judge features.
• Debugging symptoms like: "the model sometimes returns broken JSON", "the agent loops forever", "the reviewer invents issues that aren't in the code", "one bad chunk kills the whole run", "re-runs post duplicate output".
• Auditing an unfamiliar codebase for LLM call sites before changing them (use the Discovery & Audit Commands section).

Do NOT use when:

• The failure path contains no LLM — ordinary bug hunting → load debugging-playbook.
• You are choosing or pinning provider SDK versions → load dependency-management.
• You are investigating whether claims about a system are true, rather than building around a model → load research-methodology.
• You are deciding whether the LLM feature should exist at all → load engineering-minimalism first; the most reliable LLM call is the one you never make.
• You need adversarial analysis (prompt injection, data exfiltration through the model) → load security-review-playbook; this skill covers reliability, not hostile input.
• You are writing the test suite structure itself → load validation-and-testing; this skill only tells you which LLM failure paths must be covered.

Core Methodology

Work through these steps in order for every LLM integration point. Steps 1–5 are the request/response contract; 6–9 are pipeline architecture; 10–12 are the failure envelope.

1. Define a strict output contract

• In the prompt, demand JSON with an explicit shape and enum-constrained fields wherever the value drives a branch. Example contract: {"score": "correct|partial|incorrect", "explanation": "..."}.
• Show the exact shape in the prompt as a literal example, not a prose description. Models copy shapes far more reliably than they follow descriptions.
• On the parse side, validate BOTH structure and enum membership. A parse that succeeds but yields score: "mostly right" is a failure — reject it exactly like malformed JSON.
• Never branch on free text. If you find yourself writing if "correct" in response.lower(), the contract is missing; go back and add it to the prompt.

2. Put structural variation in the prompt template, not post-processing

If output shape varies by mode (multiple-choice needs an options array, free-response carries a grader-only rubric, matching pairs must be permuted), encode each mode's exact output format in a per-mode prompt template. Tell the model the shape upfront; keep the parser deterministic and single-shape per mode. Post-processing heuristics that "fix up" whichever shape came back are a smell: they grow forever, mask contract violations, and turn every new mode into parser surgery.

3. Layer JSON extraction fallbacks

Model output wrapping varies (bare JSON, fenced code block, JSON embedded in conversational chatter). Extract in strict-to-loose order and stop at the first success:

1. Parse the whole string as JSON.
2. Extract the contents of a ```json fenced block and parse that.
3. Scan for the first balanced {...} region, tracking brace depth AND string/escape state — a } inside "..." or preceded by \ does not close a brace — and parse that.

If all three layers fail, this attempt is malformed and feeds step 4. Do not add a fourth "regex repair" layer that mutates the text into parseability; loose repairs accept garbage and hide contract drift.

4. Retry with strike limits, then surface

• Malformed output → retry once with a corrective message ("Your last reply was not valid JSON. Reply with only the JSON object, nothing else.").
• Malformed twice in a row → stop retrying. Return or raise with the raw model output attached so a human (or parent agent) can see exactly what came back.
• NEVER loop until valid. An unbounded "retry until parseable" loop is an outage generator and a token furnace, and it destroys the evidence of what the model actually said.

5. Type-guard after every parse

json.loads / JSON.parse succeeding proves syntax, nothing else. json.loads('"hello"') succeeds and returns a string, not a dict. After every parse:

• Check isinstance(result, dict) (or the expected container type) before any key access.
• Check required keys exist; check enum fields against the allowed set; check value types.
• Treat a type-guard failure identically to a parse failure — it counts as a strike under step 4.

6. Inject ground truth — the model cannot invent what you hand it

For any grading, review, or judgment task, put the evidence in the prompt: the actual diff, the actual code, the actual user answer, the actual retrieved documents. The model's job is to judge supplied evidence, never to recall facts. Litmus test: if the model could plausibly answer with the evidence block deleted, your prompt invites hallucination. Corollary for UIs: attach a retrieval query/pointer to each item so evidence can be pre-fetched deterministically before the LLM call, not recalled during it.

7. Separate generation from evaluation

Generation (produce questions, produce a plan, produce review candidates) is one call whose output is cacheable and reusable. Evaluation (grade this attempt, review this diff) is per-attempt with fresh evidence each time. Never fuse them: fusing forces you to either regenerate on every attempt (cost explosion) or grade against stale evidence (correctness failure). The seam between them is a JSON contract, validated per step 5.

8. Split measurement from interpretation

Deterministic code measures; the LLM only interprets numbers it is handed.

• Anything computable without a model — pitch, onsets, cents error, latency, counts, diffs, test results — MUST be computed by code.
• Hand the LLM the measured numbers; its output is language about those numbers, never the numbers themselves.
• Join phases with JSON contracts and put a deterministic validator gate between phases: a phase's output must validate before the next phase consumes it.
• Result: the model physically cannot hallucinate a measurement, only mis-describe one — and a mis-description is auditable against the numbers on record.

9. In agent loops: tool errors are observations, not crashes

(Agent loop architecture — bounds, tool design, memory, isolation — is owned by agent-engineering; this step owns the per-call reliability mechanics inside the loop.)

When a tool the model invoked throws, catch the exception and return it to the loop as an observation string (e.g. "Error running {action}: {e}"). The model sees its mistake and can correct course on the next step. Reserve actual crashes for the fatal category in step 10. Additionally:

• Hard-cap iterations (e.g. 10 ReAct steps). On hitting the cap, return the best partial answer plus an explicit "hit iteration limit" marker — never spin.
• Apply the step-4 strike rule to the model's own malformed action JSON inside the loop, not just at the outermost call.
• Put timeouts on every tool the agent can run, sized per tool (a shell command deserves more than a math eval).

10. Classify errors: fatal vs recoverable

Build an explicit taxonomy before writing handlers, not while writing them. (The general fatal-vs-recoverable taxonomy is owned by architecture-analysis step 3; below is its LLM-path specialization.)

• Fatal (abort the whole run, fail fast): backend unreachable / connection refused, auth failure, invalid configuration. Wrap in a named exception (e.g. BackendUnavailableError) so callers can distinguish it from item-level noise. There is no point processing chunk 2 of 40 when the server is down.
• Recoverable (log + skip this unit, continue): per-item timeout, malformed JSON for one chunk, one tool failure inside a loop. One bad chunk must not kill a 40-chunk review.

Decision tree for any caught exception on the LLM path:

Exception raised on LLM path
├── Would EVERY subsequent call also fail?
│   (connection refused, bad auth, bad config)
│     └── YES → FATAL: raise named error, abort run,
│                error message names the fix ("is the backend running?")
├── Is it scoped to this one item?
│   (timeout, malformed JSON, single tool error)
│     ├── Inside an agent loop → return as observation string (step 9)
│     └── In a batch pipeline   → log with item id + raw output,
│                                 skip item, continue
└── Unknown / unclassified → treat as FATAL until classified,
                             then add it to the taxonomy

11. Bound resources

• Concurrency cap: a semaphore around concurrent LLM requests (e.g. asyncio.Semaphore(5)), sized to what the backend actually handles — local backends fall over fast under parallel load.
• Input size cap: chunk large inputs with an explicit bound (e.g. a max-lines-per-chunk constant) so no single prompt blows the context window or the latency budget.
• Timeouts on every call — per-call, not just a global default; and on every tool an agent can execute.
• Idempotency for side effects: if the pipeline posts output somewhere (PR comments, messages, files), delete or replace its own prior output before posting so re-runs never duplicate.

12. Design against LLM noise

An LLM asked to "find issues" will find issues, real or not — that is what generation means. Instruct it to report nothing when there is nothing, make "no output" a first-class success path in the code, and constrain scope in the prompt (e.g. "only genuine bugs or security issues, never style"). Silence is a valid, cheap, correct result; a pipeline that cannot produce silence produces noise.

Pre-merge checklist (run for every LLM integration point)

• Prompt shows the exact JSON shape with enum-constrained fields
• Parser validates structure + enums + types (type-guard after every parse)
• Layered extraction: whole string → fenced block → balanced braces (escape-aware)
• Strike limit: 2 consecutive malformed responses → surface raw output
• No unbounded retry / while loop anywhere on the LLM path
• Ground truth injected into every judgment prompt
• Generation and evaluation are separate calls (generation cacheable)
• All measurable quantities computed by code, not requested from the model
• Agent loops: tool errors returned as observations; hard iteration cap present
• Fatal vs recoverable taxonomy implemented, with a named fatal exception
• Concurrency semaphore + per-call timeouts + input chunk bound
• Side effects idempotent on re-run
• Tests cover: plain/fenced/embedded JSON, malformed input, strike behavior, type-guard rejection

Discovery & Audit Commands

Find every LLM call site and parsing weak point in an unfamiliar repo:

# Provider SDKs / backends in use
grep -rn -E "import (anthropic|openai|ollama)|from (anthropic|openai|ollama)|@anthropic-ai|api\.anthropic\.com|api\.openai\.com|localhost:11434" --include="*.py" --include="*.js" --include="*.ts" .

# Every JSON parse of model output (each one needs a type-guard)
grep -rn -E "json\.loads|JSON\.parse" --include="*.py" --include="*.js" --include="*.ts" .

# Retry loops that may be unbounded
grep -rn -E "while True|while \(true\)|retry|max_retries|strike" --include="*.py" --include="*.js" --include="*.ts" .

# Iteration caps in agent loops (absence is the finding)
grep -rn -E "max_iter|MAX_ITER|iteration" --include="*.py" --include="*.js" --include="*.ts" .

# Concurrency + timeout controls
grep -rn -E "Semaphore|timeout=|AbortController|signal\.alarm" --include="*.py" --include="*.js" --include="*.ts" .

# Enum validation of model output fields
grep -rn -E "allowed|enum|Enum|valid_values" --include="*.py" --include="*.js" --include="*.ts" .

# Fenced-block extraction (is layer 2 present?)
grep -rn '```json' --include="*.py" --include="*.js" --include="*.ts" .

# Named fatal exceptions for the backend
grep -rn -E "class \w*(Unavailable|Backend|Provider|Ollama)\w*Error" --include="*.py" .

# Where prompts are built (the contract lives here)
grep -rln -E "system_prompt|SYSTEM_PROMPT|prompt_template|buildPrompt|PROMPTS" .

# History of past LLM-reliability fixes (war stories = failure modes)
git log --oneline --all | grep -iE "json|retry|timeout|hallucinat|parse|llm|ollama|claude"

Audit rule: every json.loads / JSON.parse hit must be within sight of (a) an extraction fallback chain, (b) a type-guard, and (c) a bounded retry. Any hit missing one of the three is a defect, not a style issue.

Failure Modes & Anti-patterns

Symptom	Mistake	Correction
Works in dev, breaks intermittently in prod on "invalid JSON"	Parsing whole response only; no fenced/balanced-brace fallback	Layered extraction (step 3)
KeyError/TypeError crash after a successful parse	Trusting json.loads result shape	isinstance + key + enum type-guards (step 5)
Grader accepts "score": "mostly correct"	Structure validated, enum not	Reject non-enum values as a strike (step 1)
Pipeline hangs or burns tokens overnight	while not valid: retry()	2-strike limit, then surface raw output (step 4)
"Why did it fail?" is unanswerable from logs	Raw model output discarded on failure	Attach raw output to every surfaced failure (step 4)
Reviewer/grader cites code that does not exist	Model asked to judge from memory	Inject the diff/evidence into the prompt (step 6)
Cost explodes; content regenerated on every user attempt	Generation fused with evaluation	Split: generate once (cache), evaluate per attempt (step 7)
Model reports physically impossible measurements	LLM asked to measure, not interpret	Code measures; LLM interprets numbers only (step 8)
One tool exception kills a 30-step agent run	Tool errors propagate as crashes	Catch → return "Error running {action}: {e}" as observation (step 9)
Agent loops forever on a confusing task	No iteration cap	Hard cap + partial-answer return with limit marker (step 9)
Backend down → 40 log lines of per-chunk timeouts	Connection failure treated as recoverable	Named fatal exception, abort run immediately (step 10)
Local backend collapses under load	Unbounded concurrent requests	Semaphore cap sized to the backend (step 11)
Duplicate review comments pile up on every re-run	Non-idempotent side effects	Delete own prior output before posting (step 11)
Reviewer flags style nits on every clean PR	"Find issues" prompt with no null path	Scope to genuine issues; "post nothing" is success (step 12)
Parser is a growing pile of shape-specific if/else patches	Fixing output variation in post-processing	Move shape into per-mode prompt templates (step 2)

Repository Examples

Repo facts below are worked examples of the methodology — never assume your codebase matches them; re-verify before relying on any detail.

Case study: agentix — agent-loop survival (as of 2026-07-04)

~/agentix — ~700-line local ReAct agent framework on Ollama.

• Layered extraction in agentix/agent.py: (1) parse whole string, (2) extract from a ```json fence, (3) find first balanced {...} respecting string escapes — step 3's three layers, verbatim.
• Strike rule: JSON invalid twice in a row → surface raw output instead of looping. Hard cap: 10 ReAct iterations.
• Error-as-observation: tool exceptions caught and returned as "Error running {action}: {e}" observation strings so the loop survives its own tools.
• Tests mirror the failure surface: tests/test_agent.py covers plain/fenced/embedded/nested/malformed extraction, 1-strike recovery, and 2-strike surfacing — using pytest monkeypatch only, no mock framework dependency.
• Dual-model split: nomic-embed-text fixed for embeddings; chat model separately selectable — the deterministic embedding space is isolated from the swappable reasoning model.
• Tool timeouts: shell 30s, python 15s, web 15s. Build/verify: pip install -e ".[dev]" then pytest (hatchling build).

Case study: NYTW — contract-first grading (as of 2026-07-04)

~/NYTW — Next.js 16 + Node CLI quiz tool on the Anthropic Claude API.

• frq.js builds grading prompts that inject the actual git diff + codebase evidence + user answer (ground-truth injection, step 6) and request strict JSON {"score":"correct|partial|incorrect","explanation"}; parsing validates against the allowed enum values (step 1).
• questions.js: mode variation (MCQ needs options, FRQ carries a grader-only rubric, matching pairs permuted) lives in MODE_FORMATS prompt templates — the LLM is told the exact output shape upfront, so parsing is deterministic; no post-processing heuristics (step 2).
• Generation (one LLM call, cacheable) is separated from grading (one call per attempt, fresh evidence) — step 7 verbatim.
• Each question carries a perseus_query so the UI pre-fetches codebase evidence deterministically before grading. Tests: npm test from quiz/ (Node native test runner, no framework added).

Case study: prism — backend error taxonomy (as of 2026-07-04)

~/prism — FastAPI AI PR reviewer on local Ollama.

• Fatal vs recoverable in prism/reviewer.py: connection failure → custom OllamaUnavailableError aborts the entire review (fail fast); timeout or invalid LLM JSON → log + skip that chunk only, continue (step 10).
• isinstance type-guards after every json.loads (step 5). asyncio.Semaphore(5) caps concurrent LLM requests; chunking bounded by MAX_LINES_PER_CHUNK=120 (step 11).
• Noise design: reviews only genuine bugs/security issues and posts nothing when there are none (step 12). Deletes its own prior PR comments before posting — idempotent re-review (step 11).
• Hardening arrived as deliberate commits: lineage e264a72 → b7bbabb → 51b92c9 "security: add payload size limit, concurrency cap, repo name validation, Ollama timeout" → b011239 "tests: add security path coverage for signature, repo validation, size limit". Tests are risk-first: pytest tests/ runs tests/test_security.py only. Install: pip install -e ..

Case study: orphy — measurement/interpretation split (as of 2026-07-04)

~/orphy — Vite+React vocal coach.

• DSP layer measures (pitch, onsets, cents error); LLM layer only interprets those numbers as language — the model cannot hallucinate a measurement (step 8).
• Phases joined by JSON contracts, with deterministic validators gating each phase.
• FeedbackResult decouples generation from delivery: DeliveryChannel mode text|audio is swappable with no rewrite. Run: npm run dev / npm run build / npm run preview.

Validation Criteria

You applied this skill correctly when:

1. Every LLM call site in the diff passes the pre-merge checklist, item by item, not by vibes.
2. For each call site you can answer "what happens when the model returns garbage twice?" — and the answer is "raw output is surfaced within N bounded attempts", never "it retries".
3. Feeding a deliberately malformed response (via test or manual patch) produces a clean, raw-output-bearing failure — not a crash and not a loop.
4. Killing the backend mid-run produces exactly one named fatal error — not per-item error spam.
5. A judgment prompt with its evidence block deleted would be unanswerable by the model.
6. No number the code could compute is being requested from the model.
7. Running the pipeline twice in a row produces no duplicated side effects.
8. The test suite fails if any extraction layer, strike rule, or type-guard is removed.

Provenance & Maintenance

• Sources: ~/agentix, ~/NYTW, ~/prism, ~/orphy — investigated 2026-07-04. Owner confirmed (2026-07-04) that LLM output reliability is their hardest recurring problem and that this skill should be exhaustive and operational. The methodology is generalized from these repos; repo details are examples only, never assumptions baked into the method.
• Assumptions: file paths (agentix/agent.py, prism/reviewer.py, frq.js, questions.js) and constants (Semaphore(5), MAX_LINES_PER_CHUNK=120, 10-iteration cap, 2-strike rule, tool timeouts 30s/15s/15s) are point-in-time facts. Any detail not listed in the case studies above is (hypothesis — requires verification).
• Re-verification commands:

  cd ~/agentix && grep -n "json" agentix/agent.py | head -30 && pytest -q
  cd ~/prism && git log --oneline -6 && grep -rn "Semaphore\|MAX_LINES_PER_CHUNK\|OllamaUnavailable" prism/
  cd ~/NYTW && grep -rn "MODE_FORMATS\|perseus_query" --include="*.js" . | head
  

• Likely to drift: numeric constants (semaphore size, chunk bound, iteration cap, timeouts); model names (nomic-embed-text); Anthropic API request shapes; Next.js/Node versions; prism commit tips beyond b011239.
• Maintenance checklist:
  • Re-run the re-verification commands; update constants and paths that moved.
  • Re-stamp "(as of ...)" dates on all case studies after re-checking.
  • When a new repo adds an LLM integration, mine it for a new failure-mode table row before adding a case study.
  • Confirm cross-referenced skills (debugging-playbook, security-review-playbook, engineering-minimalism, research-methodology, dependency-management, validation-and-testing) still exist under those directory names.