Llm app code

LLM App Code

Code around an LLM has one governing fact: the model is an untrusted, non-deterministic input source that reads like a trusted colleague. Most LLM-app bugs come from forgetting one half of that sentence.

Model output is untrusted input — always

• Parse defensively. Every JSON.parse of model output wrapped and shape-validated; expect markdown fences around JSON, trailing prose, truncated output at max_tokens, and empty strings. A malformed model response must degrade a turn, never crash the app.
• Validate tool-call arguments against the schema server-side before execution — required keys, types, enums, AND rejection of unexpected keys. The model will eventually send every wrong shape.
• Whitelist strict-enum values headed to external systems. Model-generated slugs/networks/plans get mapped through a server-side table — never passed through, even when the prompt "guarantees" the format.
• Authorization never derives from model output. The model can propose; only server-recorded state + explicit user action can approve. Never let a model-visible tool perform destructive/financial writes directly — gate writes behind a human-approval step bound server-side (proposal pattern). Never trust a model's claim about what the user consented to.
• Prompt injection is transitive: any tool result containing third-party text (web pages, tickets, emails) is a channel for instructions to your model. Treat tool results as data in your prompt design; never give a model simultaneously (a) exposure to untrusted text and (b) unsupervised dangerous capabilities.

Determinism where it matters

Route load-bearing guarantees through code, not prompts: format enforcement, redaction/scrubbing of sensitive figures, filtering which tools the model sees, output length caps. A prompt instruction is a strong suggestion; a deterministic post-processor is a guarantee. Use both (prompt = primary, code = backstop).

Prompt & model changes are behavior changes

• Never ship a prompt/model change on vibes. Run the project's eval suite; if none exists for the behavior you're changing, write at least a smoke set first. A model swap that "just works" in one manual test can regress safety/format behaviors discovered only under repetition.
• Cache economics: providers cache the prompt prefix. Keep stable content (system prompt, tool defs) first and byte-identical across turns; put variable content last. A "harmless" reordering or timestamp in the system prompt can silently multiply token cost.
• Pin model IDs explicitly; know each pinned model's parameter quirks (thinking/effort defaults differ per model — verify, don't assume compatibility when swapping).

Streaming and the agent loop

• Handle the full event grammar: partial deltas, tool-use blocks interleaved with text, stop reasons (max_tokens mid-JSON!), and error events mid-stream. Test the abort path — client disconnects mid-generation must clean up.
• Agent loops need: an iteration cap, per-tool timeouts, token/cost budget, and dedupe/idempotency on tool execution (retried turns re-issue tool calls).
• Retries: retry transient API errors with backoff; NEVER blind-retry a turn whose tool side-effects may have executed.

Cost/latency hygiene

Log tokens + cost per turn from day one (you cannot fix what you don't attribute). Trim tool results before feeding them back (models don't need 400-row payloads). Choose the model tier per call site deliberately — classification/extraction rarely needs the flagship.