agentsclimarketplace

Otel span verifier

Skill seanchatmangpt/ggen/.claude/skills/otel-span-verifier

Verify OTEL spans exist for LLM calls, MCP tools, and pipeline stages. Mandatory after any external service integration. Tests passing is not sufficient.From its SKILL.md

Install
npx -y skills add seanchatmangpt/ggen --skill otel-span-verifier

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 5 stars5 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

3.0 KB, as published. Nobody here has run it

Skill: otel-span-verifier

Doctrine

Tests passing is not proof. OTEL spans are proof.

A test can pass while mocking the LLM call, stubbing the MCP tool, or skipping the external service entirely. Passing tests prove the test harness works. OTEL spans prove the external service was called.

When to Trigger

  • After implementing any LLM integration (Groq, OpenAI, Anthropic, etc.)
  • After adding MCP tool execution
  • After implementing pipeline stages (μ₁-μ₅)
  • After external API call integration
  • User claims LLM/external service feature is "complete" — verify first

Verification Procedure

Step 1: Enable Trace Logging

export RUST_LOG=trace,ggen_ai=trace,ggen_core=trace,genai=trace

Step 2: Run Relevant Test

cargo test -p ggen-cli-lib --test llm_e2e_test -- --nocapture 2>&1 | tee otel_output.txt

Step 3: Grep for Required Spans

# LLM integration
grep -E "llm\.complete|llm\.complete_stream" otel_output.txt
grep -E "llm\.model" otel_output.txt
grep -E "llm\.(prompt_tokens|completion_tokens|total_tokens)" otel_output.txt

# MCP tools
grep -E "mcp\.tool\.call|mcp\.tool\.response" otel_output.txt
grep -E "mcp\.tool\.name" otel_output.txt

# Pipeline stages
grep -E "pipeline\.(load|extract|generate|validate|emit)" otel_output.txt
grep -E "pipeline\.stage|pipeline\.duration_ms" otel_output.txt

Step 4: Verify Attributes

Real evidence has:

  • ✅ Non-zero token counts (not mocked)
  • ✅ Real network latency (2-3s for LLM calls, not instant)
  • ✅ All required attributes populated
  • ✅ Correct model names (e.g., groq::openai/gpt-oss-20b)

Required Spans by Feature

FeatureRequired SpansAttributes
LLMllm.complete, llm.complete_streamllm.model, llm.prompt_tokens, llm.completion_tokens, llm.total_tokens
MCP Toolsmcp.tool.call, mcp.tool.responsemcp.tool.name, mcp.tool.duration_ms
Pipelinepipeline.load, pipeline.extract, pipeline.generate, pipeline.validate, pipeline.emitpipeline.stage, pipeline.duration_ms
Quality Gatesquality_gate.validate, quality_gate.pass_failgate.name, gate.result

Failure Modes

ObservationConclusion
Spans exist with non-zero tokens and 2-3s latency✅ PROVEN — Real LLM call
No spans in output❌ UNVERIFIED — Test may be mocked
Spans with zero tokens or <100ms latency⚠️ OBSERVED — Check if synthetic response
Span without llm.model attribute⚠️ PARTIAL — Call initiated, response data missing

Before Claiming Completion

  1. ✅ All tests pass
  2. ✅ OTEL spans exist for the operation
  3. ✅ All required attributes populated
  4. ✅ Token counts and latency realistic
  5. ✅ Error spans appear if operation failed

No spans = feature not done. Non-negotiable.

Keep looking

Skills are one crate of 610,220. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.