agentsclimarketplace

Instrument llm observability

Skill ContextJet-ai/awesome-llm-observability/skills/instrument-llm-observability

50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

Install
npx -y skills add ContextJet-ai/awesome-llm-observability --skill instrument-llm-observability

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.

What its author says it does

Copied from the file, not written here

Use this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step. Trigger whenever someone wants to "add tracing", "instrument", "monitor", "see what my agent is doing", or debug an LLM app in production. Prefer vendor-neutral OpenTelemetry unless a specific platform is already in use.

The file declares its own license as CC0-1.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

3.5 KB, as published. Nobody here has run it

Instrument an LLM app for observability

Add production-grade tracing to an LLM/agent app so every run is inspectable: prompts, tool calls, retrievals, token usage, latency, and cost per span.

Decide the approach first (2 questions)

  1. Is a backend already chosen? (Langfuse, Arize Phoenix, Comet Opik, Datadog, Grafana…). If yes, use its SDK/auto-instrumentation. If no, default to OpenTelemetry GenAI semantic conventions so traces export to any OTel backend and you avoid lock-in.
  2. Lowest-friction path available? If the app calls LLMs through a gateway/proxy (LiteLLM, Helicone, Portkey), you may get tracing + cost with zero code by changing the base URL. Check that before adding an SDK.

Vendor-neutral path (recommended default): OpenTelemetry

Emit gen_ai.* spans via an OTel instrumentation library, then point at any collector.

  • Use OpenLLMetry (traceloop-sdk) or OpenInference (Arize) - both emit OTel-compatible LLM spans.
  • Minimal shape:
    1. Install the instrumentation lib + an OTLP exporter.
    2. Initialize once at startup (set service name + OTLP endpoint via OTEL_EXPORTER_OTLP_ENDPOINT).
    3. Auto-instrument the LLM SDK (OpenAI/Anthropic/etc.); wrap custom agent steps in manual spans (gen_ai.operation.name, tool spans).
  • Result: spans carry model, prompt/completion, token counts, latency; cost is derived downstream.

See references/opentelemetry.md for the exact span attributes to set and a copy-paste init.

Platform SDK path (if a backend is already picked)

Wrap the LLM client with the platform's tracer (e.g. track_openai(...), @observe, an OTel exporter to that backend). Prefer their auto-instrumentation over hand-rolled spans; add manual spans only for custom tool/retrieval steps the SDK can't see.

Instrument the right things (checklist)

  • Each LLM call: model, input messages, output, prompt_tokens/completion_tokens, latency.
  • Tool/function calls: name, args, result, error.
  • Retrieval steps (RAG): query, retrieved doc IDs/scores.
  • Agent/chain boundaries: one parent span per user request, child spans per step.
  • Errors: record exceptions on the span (don't swallow them - a silent except that returns empty output hides the failure).
  • Cost: derive from token counts × model price (most platforms do this for you).

Verify

Run one representative request and confirm the trace tree shows the full chain (parent → LLM/tool/retrieval children) with tokens + latency populated. If spans are missing, the instrumentation isn't wrapping that code path - instrument it manually.

Anti-patterns

  • Logging raw prints instead of structured spans (no trace tree, no cost rollup).
  • Instrumenting only the top-level call, missing tool/retrieval steps (the interesting failures hide there).
  • Hard-coding a vendor SDK when OTel would keep you portable.
  • Sending full prompts/PII to a third-party backend without redaction - mask sensitive fields first.

Tool picker

Not sure which backend? See the sibling choose-observability-stack skill, or the curated list in this repo's README.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.