Agent observability

Skill m00kk/agent-skills-playbook/skills/agent-observability

Adds tracing, logging, metrics, and cost tracking for LLM agents and LangGraph workflows. Use when debugging agents, LangSmith, OpenTelemetry, production monitoring, or latency/cost optimization.From its SKILL.md

Install

npx -y skills add m00kk/agent-skills-playbook --skill agent-observability

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.5 KB, 326 tokens by cl100k_base, as published. Nobody here has run it

Agent Observability

What to instrument

Signal	Examples
Traces	LLM calls, tool calls, graph node transitions
Metrics	latency p95, tokens in/out, tool error rate
Logs	run_id, thread_id, tool name (no secrets)
Cost	$ per run, per tool, per user tier

Workflow

- [ ] Assign stable run_id / thread_id per session
- [ ] Wrap each LLM call with span (model, tokens, latency)
- [ ] Wrap each tool call with span (name, status, duration)
- [ ] Aggregate token usage at end of run
- [ ] Alert on error rate or cost anomaly thresholds

LangSmith (LangChain/LangGraph)

Set env: LANGCHAIN_TRACING_V2=true, LANGCHAIN_API_KEY, LANGCHAIN_PROJECT
Tag runs: environment, feature, user_id (hashed)

OpenTelemetry

Use GenAI semantic conventions where supported
Export to vendor backend (Honeycomb, Datadog, etc.)
Correlate trace_id across MCP server + orchestrator

Dashboards (minimum)

Runs per hour + success rate
Token usage by model
Top 5 slowest tools
Top 5 failing tools

Privacy

Scrub PII from prompts in stored traces or use retention limits
Hash user identifiers in labels

See evaluate-agent-quality for regression datasets tied to traces.

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.