Trace multi agent system
Skill ContextJet-ai/awesome-llm-observability/skills/trace-multi-agent-system
Use this to add observability to a multi-agent or agentic system (multiple agents, sub-agents, tool loops, handoffs). Trigger on "trace my agents", "my agent crew is a black box", "which agent failed", "debug my LangGraph/CrewAI/AutoGen/agent workflow", "the agents loop forever". Get a clear span tree across agents and tool calls so you can see who did what.From its SKILL.md
npx -y skills add ContextJet-ai/awesome-llm-observability --skill trace-multi-agent-systemAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
What its file declares
Copied from the file, not written here
The file declares its own license as CC0-1.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.
SKILL.md
3.0 KB, 612 tokens by cl100k_base, as published. Nobody here has run it
Trace a multi-agent system
Multi-agent systems fail in ways single calls do not: an agent hands off bad state, a tool loop never terminates, a sub-agent silently fails and the orchestrator carries on. You cannot debug that from the final output. You need the full tree of who called whom.
Get the structure right
Model the run as a nested span tree:
- One root span per user request or task.
- One span per agent turn (which agent, its role, its input state).
- Child spans for each tool call, retrieval, and LLM call inside that turn.
- Handoff spans / events when one agent passes control or state to another (record what was passed).
Most agent frameworks emit this if you turn on their tracing (LangGraph, CrewAI, AutoGen, OpenAI Agents SDK, LlamaIndex) or add OpenTelemetry instrumentation (OpenLLMetry, OpenInference auto-instrument several agent frameworks). Prefer the framework's built-in tracing, then fill gaps with manual spans (see instrument-llm-observability).
What to capture per agent
- Identity + role (
agent.name,agent.role) so spans are attributable. - Input state and output at each handoff (the #1 source of multi-agent bugs is corrupted or lost state between agents).
- Tool calls with args + results + errors.
- Loop/iteration count so runaway loops are visible.
- Tokens + latency + cost per agent (rolled up to the whole run) so you can see which agent is the expensive/slow one.
Debug the common failures
| Symptom | Look at |
|---|---|
| Wrong final result | Walk the tree to the first agent whose output diverges from intent. Fix there. |
| Infinite / long loops | Iteration counts + repeated identical tool spans. Add a max-iterations guard. |
| One agent "did nothing" | A missing or errored subtree. A swallowed exception in a sub-agent. |
| Blows the budget | Per-agent token rollup. Usually one agent re-sending full context each turn. |
| Non-deterministic flakiness | Compare two traces of the same input side by side. |
Verify
- Run one task and confirm the trace shows every agent turn, every tool call, and every handoff, nested correctly.
- Confirm per-agent cost + latency roll up to a run total.
- Break a sub-agent on purpose and confirm the failure is visible in the tree (not swallowed).
Anti-patterns
- Logging only the orchestrator, so sub-agent behavior is invisible.
- No handoff/state capture, so you cannot tell where state got corrupted.
- No iteration count, so a runaway loop just looks like "it is slow and expensive."
- Treating a multi-agent bug as a prompt bug when it is really a state-passing bug between agents.
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.
Gives 0 of the 12 instructions most agent orchestration skills give in 612 tokens
Counted across 848 of the 1,300 authors here whose files we hold, read 2026-09-06
- Dispatch one agent per independent problem domainin 56 of 848, across 42 files
- Run full test suite after integrationin 55 of 848, across 42 files
- Verify fixes do not conflictin 40 of 848, across 32 files
- Review each summary when agents returnin 40 of 848, across 31 files
- Write a handoff document summarising the current conversationin 30 of 848, across 25 files
- Reference existing artifacts by path or URLin 26 of 848, across 24 files
- Give each agent a specific scopein 19 of 848, across 10 files
- Give each agent a clear goalin 19 of 848, across 10 files
- Include a suggested skills section in the documentin 18 of 848, across 16 files
- Tailor the doc to the user argumentsin 18 of 848, across 15 files
- Issue all subagent dispatches in the same responsein 17 of 848, across 11 files
- Use git worktrees for isolationin 17 of 848, across 8 files
Said here and by no other author read
- Model the run as a nested span tree
- Capture identity and role per agent
- Record input state and output at each handoff
- Capture tool calls with args, results, and errors
- Roll up tokens, latency, and cost per agent
- Run one task and confirm the trace shows every agent turn
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.