Debug agent from traces
Skill ContextJet-ai/awesome-llm-observability/skills/debug-agent-from-traces
50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.
npx -y skills add ContextJet-ai/awesome-llm-observability --skill debug-agent-from-tracesAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
What its author says it does
Copied from the file, not written here
Use this to diagnose WHY an LLM agent or chain produced a wrong, empty, slow, or expensive result, by reading its observability trace. Trigger on "my agent gave the wrong answer", "the chain returned nothing", "why is this so slow/expensive", "debug this trace/run", or when a trace tree is available. Walk the span tree systematically instead of guessing.
The file declares its own license as CC0-1.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.
SKILL.md
3.1 KB, 663 tokens by cl100k_base, as published. Nobody here has run it
Debug an agent from its trace
A trace is a tree of spans (parent request → LLM/tool/retrieval children). Most agent failures are visible in it if you read it in the right order. Don't guess from the final output - walk the tree.
Triage by symptom
| Symptom | Look here first |
|---|---|
| Wrong answer | The LLM span just before the bad output: was the prompt/context correct? Then the retrieval span that fed it. |
| Empty / truncated output | finish_reason (length? content_filter?), max_tokens, and any span that raised an exception then got swallowed. |
| Hallucinated facts | Retrieval spans - were the right docs retrieved (check scores/IDs)? If not, it's a retrieval bug, not an LLM bug. |
| Too slow | Span durations - find the critical path. Usually one slow retrieval, a serial loop that should be parallel, or a huge context. |
| Too expensive | Token counts per LLM span - find the span with the largest input_tokens (usually bloated context or a retry storm). |
| Silent failure | A span with an error/exception that was caught and returned empty. A missing subtree = a step that never ran. |
The systematic walk
- Start at the root, confirm the user input is what you expect.
- Follow to the first LLM call. Read the actual rendered prompt + context (not the template). Most "model is dumb" bugs are actually "we fed it the wrong context".
- Check each tool/retrieval span in order: inputs correct? output sane? error?
- Find the divergence point - the first span where reality differs from intent. Fix there, not at the output.
- Check token + latency on every LLM span to catch cost/perf issues even when the answer is right.
Common root causes (in order of frequency)
- Wrong/empty retrieved context → the LLM was set up to fail. (Retrieval bug.)
- Prompt/template rendering bug → a variable didn't interpolate; context is blank or duplicated.
- Swallowed exception → a
try/exceptthat logs and returns empty, dropping context or output silently. Grep for these. - Retry storm → the same call repeated N times (rate limit / transient error) inflating cost + latency.
- Context bloat → the whole history re-sent each turn;
input_tokensgrows every step.
Turn the fix into a regression test
Once you find the divergence, capture that input as an eval case (see the add-llm-evals skill) so the bug can't come back. Debugging a trace and not adding a test means you'll debug it again next month.
Anti-patterns
- Re-prompting the model when the real bug is upstream retrieval/rendering.
- Reading only the final output and inferring the cause.
- Fixing the symptom without adding a regression test.