Add llm fallbacks

Skill ContextJet-ai/awesome-llm-observability/skills/add-llm-fallbacks

50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

Install

npx -y skills add ContextJet-ai/awesome-llm-observability --skill add-llm-fallbacks

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.

What its author says it does

Copied from the file, not written here

Use this to make an LLM app resilient to provider failures, rate limits, timeouts, and outages. Trigger on "handle LLM API errors", "add retries/fallbacks", "the app breaks when OpenAI is down", "rate limit errors", "make my LLM calls reliable", "timeout handling". Add retries, timeouts, and model/provider fallbacks, and observe them so failures are visible.

The file declares its own license as CC0-1.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

3.3 KB, as published. Nobody here has run it

Add LLM fallbacks and resilience

LLM providers rate-limit, time out, 500, and occasionally go down. A single unguarded call means your app goes down with them. Add layered resilience, and instrument it so you can see when it kicks in.

The layers (add all of them)

Timeouts. Every call needs a sane timeout. An unbounded call can hang a request forever. Set it based on your p95 latency plus headroom.
Retries with backoff. Retry transient failures (429, 500, 503, timeouts) with exponential backoff + jitter. Cap the number of retries and cap the total wait, and honor Retry-After when present. A naive retry loop turns a rate limit into a retry storm (and a big bill, see reduce-llm-cost).
Do not retry non-transient errors. 400s (bad request), auth errors, and content-policy refusals will fail again. Retrying them wastes time and money.
Model / provider fallback. If the primary model errors or is down, fall back to a secondary (same provider different model, or a different provider). A gateway (LiteLLM, Portkey) gives you this with config instead of code.
Graceful degradation. When everything fails, return a sensible fallback (cached answer, a "try again shortly" message, a simpler non-LLM path) instead of a stack trace.

Instrument it (so resilience is not invisible)

The failure mode of good resilience is that it hides problems. Emit telemetry so you still see them:

A span/event when a retry happens (and how many).
A span/event when a fallback fires (primary -> secondary).
Error rate per provider/model and fallback rate on a dashboard, with alerts (see set-up-drift-alerts).

If your fallback rate silently climbs to 30%, you want to know, not find out from the bill or a user.

Watch the cost/quality of fallbacks

A fallback model may be cheaper-but-worse or pricier. Track cost and (ideally) quality on the fallback path too, so a provider outage does not silently degrade your app for hours.

Verify

Kill the primary provider (bad key / block the host) and confirm the app fails over, not over.
Simulate a 429 and confirm backoff + Retry-After are honored, with a capped total wait.
Confirm retries and fallbacks show up in traces/metrics.

Anti-patterns

No timeout (one hung call ties up a worker).
Retrying 4xx/auth/policy errors (they will not succeed; you just wait and pay).
Fixed-delay retries with no cap (retry storm on a rate limit).
Silent fallbacks with no telemetry (you are degraded and blind to it).
A fallback model nobody evaluated (you fail over into worse answers).

Grounding

Standard distributed-systems resilience patterns applied to LLM calls: exponential backoff with jitter (AWS Architecture guidance), retry/timeout/circuit-breaker (Nygard, Release It!). Gateways implement provider fallback: LiteLLM, Portkey.