Multi agent orchestration

Overview

Single agents are limited by context window, specialization depth, and parallelism. Multi-agent systems overcome these limits by routing subtasks to specialized agents. But multi-agent systems introduce new failure modes: lost context, conflicting decisions, infinite loops, and cascading failures.

This skill provides the architecture and coordination patterns to build multi-agent systems that are reliable, observable, and maintainable.

When to Use

• The task requires more context than a single agent can handle
• Different subtasks require different specializations (research, coding, review, security)
• Subtasks can be parallelized for speed
• The workflow is long-running and requires checkpointing
• Different tasks require different levels of human oversight

Process

Step 1: Design the Agent Network

1. Define agent responsibilities: Each agent should have a single, well-defined job. Name them by role: researcher, coder, reviewer, security-auditor, tester.
2. Define communication topology: Who can talk to whom?
   • Pipeline: Agent A → Agent B → Agent C (sequential)
   • Supervisor: Orchestrator dispatches to specialists (hub-and-spoke)
   • Peer: Agents collaborate as equals (mesh)
3. Define data contracts: What does each agent receive? What does it output? Use structured formats (JSON schemas) for inter-agent communication.
4. Define the orchestration logic: Who decides which agent acts next?

Verify: You can draw the agent network on a whiteboard with clear roles and data flow.

Step 2: Implement Context Management

5. Each agent should receive only the context it needs — not the full conversation history.
6. Use a shared state store (database, key-value store) for information that multiple agents need.
7. Pass summaries, not full transcripts, when context must traverse agent boundaries.
8. Include a task ID in every message for tracing.

Verify: No agent receives more context than it requires for its specific task.

Step 3: Design for Failure

9. Every agent call can fail — plan for it:
   • Timeout with a defined maximum duration
   • Retry with exponential backoff (max 3 retries)
   • Fallback behavior when retries are exhausted
10. Prevent infinite loops: Track call depth. If depth > N (e.g., 10), surface to human review.
11. Checkpointing: For long workflows, save state after each major step so the workflow can be resumed after failure.
12. Dead letter queue: Failed tasks that exhaust retries go to a queue for human inspection.

Verify: Failure scenarios are defined for every agent-to-agent call.

Step 4: Human-in-the-Loop Checkpoints

13. Define which decisions require human approval:
    • Irreversible actions (data deletion, financial transactions, external communications)
    • High-uncertainty states (agents disagree, confidence below threshold)
    • Sensitive operations (PII access, privileged system access)
14. Design the human review interface: What information does the reviewer need? What actions can they take?

Verify: At least one human-in-the-loop checkpoint exists for high-risk operations.

Step 5: Observability

15. Log every agent invocation: inputs, outputs, duration, token usage, errors.
16. Implement distributed tracing across the agent network (trace ID propagated through all calls).
17. Dashboard: agent activity, success/failure rates, latency, token consumption.
18. Alerts: agent down, retry rate spike, context overflow, unexpected output patterns.

Verify: You can trace any specific task's full execution path across all agents from logs alone.

Common Rationalizations (and Rebuttals)

Excuse	Rebuttal
"One agent is simpler"	Until it hits context limits, fails silently, or produces wrong results. Multi-agent is the right tool for complex tasks.
"We'll add observability later"	Multi-agent systems without observability are black boxes. Debug them in production — I dare you.
"Agents are smart, they'll figure it out"	Agents are tools. They need clear roles, contracts, and failure boundaries.
"The happy path works fine"	Multi-agent systems fail in complex ways. Design for failure from day one.

Red Flags

• Agents pass full conversation history to other agents (context bloat)
• No timeout defined for any agent call
• Agents can call each other recursively without depth limits
• No human approval required for irreversible actions
• No distributed tracing across agent boundaries
• Agents making conflicting state changes with no conflict resolution

Verification

• Agent network designed with clear roles and data contracts
• Context is minimized at each agent boundary
• Failure handling (timeout, retry, fallback) for every agent call
• Infinite loop prevention via call depth limits
• Human-in-the-loop checkpoints for high-risk operations
• Distributed tracing implemented across agents
• End-to-end test of failure scenarios

References

• task-decomposition skill
• observability skill
• prompt-injection-defense skill
• hallucination-prevention skill