Audit langfuse llm

Langfuse LLM Quality Audit

Automated PDCA audit for LLM/AI features: trace completeness, prompt quality, cost efficiency, eval health, grounding accuracy, and end-to-end pipeline verification. Works with any project — auto-detects Langfuse setup from the codebase.

Critical Rules

NEVER skip the auto-detect phase. Every project configures Langfuse differently.

Research before judging. Use Firecrawl to find current LLM best practices so recommendations are evidence-based, not opinion.

Verify live, not just statically. Trigger AI features via Playwright and confirm traces land in Langfuse — static code analysis alone misses runtime issues.

Use concrete numbers. "Costs seem high" is not an audit finding. "gpt-4.1 used for intent classification at $0.02/call when gpt-4.1-mini at $0.002/call achieves equivalent accuracy" is.

Always use the browser-anti-stall protocol when using Playwright browser MCP tools.

---

Phase 0: Auto-Detect Langfuse Integration

0a. Find Langfuse Configuration

Search for environment variables and config files (in order):

1. .env, .env.local, .env.production — look for LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_BASE_URL, LANGFUSE_HOST
2. langfuse.config.ts, langfuse.config.js — dedicated config files
3. instrumentation.ts / instrumentation.js — Next.js instrumentation with Langfuse
4. Supabase Edge Functions — Glob("**/supabase/functions/**/index.ts") and search for Langfuse imports

Grep(pattern: "LANGFUSE_PUBLIC_KEY|LANGFUSE_SECRET_KEY|LANGFUSE_BASE_URL|LANGFUSE_HOST", glob: ".env*")
Grep(pattern: "langfuse|Langfuse|@langfuse", glob: "*.{ts,js,tsx,jsx,py,rb,go}")

Record:

• LANGFUSE_HOST (cloud or self-hosted URL)
• LANGFUSE_PUBLIC_KEY (identifies the project)
• Which source files import/use Langfuse
• Whether the CLI env vars are available (the Shell commands below require LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY in the environment)

0b. Detect LLM Framework and Provider

Grep(pattern: "openai|OpenAI|anthropic|Anthropic|@google/generative-ai|gemini|cohere|mistral|groq|together|replicate", glob: "*.{ts,js,py}")
Grep(pattern: "langchain|LangChain|@langchain|vercel/ai|ai/core|createOpenAI|createAnthropic", glob: "*.{ts,js,py}")

Record:

• LLM providers (OpenAI, Anthropic, Google, etc.)
• LLM frameworks (LangChain, Vercel AI SDK, direct API calls, etc.)
• Model names used (grep for model name strings like gpt-4.1, claude-opus-4-8, gemini-2.5-pro)

0c. Map AI Features

SemanticSearch(query: "Where are LLM/AI features called in the codebase?", target_directories: [])

Build a feature map:

Feature	File(s)	Provider	Model	Traced?
e.g. Chat	app/api/chat/route.ts	OpenAI	gpt-4.1	Yes

0d. Detect Eval and Prompt Management Setup

Grep(pattern: "createScore|langfuse.score|annotation|eval|judge|dataset", glob: "*.{ts,js,py}")
Grep(pattern: "getPrompt|langfuse.prompt|fetchPrompt|compilePrompt", glob: "*.{ts,js,py}")

Record:

• Prompt management approach: Langfuse managed prompts vs hardcoded vs config file
• Eval setup: annotation queues, programmatic scoring, judge LLM, dataset runs
• Whether prompts are versioned and labeled

---

Phase 1: Research LLM Best Practices

Before auditing, establish the current state of the art so findings are grounded in evidence.

1a. Firecrawl Research

firecrawl:firecrawl_search
{
 "query": "LLM observability best practices production monitoring [current year]",
 "limit": 5
}

firecrawl:firecrawl_search
{
 "query": "prompt engineering evaluation scoring hallucination detection [current year]",
 "limit": 5
}

firecrawl:firecrawl_search
{
 "query": "LLM cost optimization token usage model selection production [current year]",
 "limit": 5
}

Scrape the top 2-3 most relevant results for detailed guidance:

firecrawl:firecrawl_scrape
{
 "url": "<BEST_RESULT_URL>",
 "formats": ["markdown"]
}

1b. Langfuse Documentation

Research Langfuse-specific features relevant to the detected setup:

firecrawl:firecrawl_search
{
 "query": "site:langfuse.com docs tracing prompts evaluation scores",
 "limit": 5
}

If the project uses a specific LLM framework (LangChain, Vercel AI SDK, etc.), also fetch its Langfuse integration docs.

1c. Context7 for LLM Framework Docs

If detected in Phase 0b, fetch the framework-specific documentation:

context7:resolve-library-id
{
 "libraryName": "<DETECTED_FRAMEWORK e.g. langchain or vercel-ai>"
}

Then query for integration patterns:

context7:query-docs
{
 "libraryId": "<RESOLVED_ID>",
 "query": "Langfuse integration tracing observability"
}

---

Phase 2: Audit via Langfuse CLI

All commands below use the Langfuse CLI via the Shell tool. Ensure the environment has LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY set (from .env or exported).

2a. Trace Completeness

npx langfuse-cli api traces list --limit 50

For each AI feature identified in Phase 0c, verify:

• Trace exists with a matching name/metadata
• Trace has spans/generations (not just a top-level trace with no children)
• Trace includes input/output (not empty)
• Trace has proper metadata (userId, sessionId, tags)
• Latency is recorded

Red flags:

• AI feature exists in code but produces no traces → missing instrumentation
• Traces exist but have no generations → incomplete tracing (wrapper created but LLM call not captured)
• Traces with empty output → output not being captured (fire-and-forget pattern)

2b. Prompt Quality Audit

npx langfuse-cli api prompts list

For each prompt:

npx langfuse-cli api prompts get --name "<PROMPT_NAME>"

Evaluate:

• Versioning: Are prompts versioned (v1, v2, v3+) or stuck at v1?
• Labels: Is there a production label? Are there staging/experiment labels for A/B testing?
• System message quality: Clear role definition, constraints, output format instructions?
• Few-shot examples: Does the prompt include examples for complex tasks?
• Guardrails: Does the prompt include instructions to refuse off-topic/harmful requests?
• Variables: Are dynamic parts properly templated with {{variables}} not string concatenation?
• Freshness: When was the prompt last updated? Stale prompts may not use newer model capabilities.

If prompts are hardcoded in source code instead of managed via Langfuse:

Grep(pattern: "You are|system.*message|systemPrompt|SYSTEM_PROMPT", glob: "*.{ts,js,py}")

Flag hardcoded prompts as a finding — they should be migrated to Langfuse for versioning and A/B testing.

2c. Model and Cost Efficiency

From trace data, analyze:

npx langfuse-cli api traces list --limit 50

For each trace, check the generation details (model, usage tokens, latency, cost).

Build a cost table:

Feature	Model	Avg Input Tokens	Avg Output Tokens	Avg Latency	Est. Cost/Call

Red flags:

• Expensive model (gpt-4.1, claude-opus-4-8) used for simple classification/extraction → recommend cheaper model (e.g. gpt-4.1-mini, claude-haiku-4-5)
• High input token counts → check for unnecessary context stuffing
• Output tokens much larger than needed → add max_tokens or response format constraints
• High latency on user-facing features → consider streaming, caching, or smaller model
• Same content sent repeatedly → implement semantic caching

2d. Eval Score Health

npx langfuse-cli api scores list --limit 50

Evaluate:

• Score existence: Are evals running at all?
• Score types: What's being measured (relevance, faithfulness, toxicity, custom)?
• Score distribution: Are scores clustered (all 1.0 = useless eval) or distributed?
• Annotation queues: Are humans reviewing AI outputs?
• Judge LLM: If using LLM-as-judge, which model? Is the judge prompt well-designed?

Red flags:

• No scores at all → no quality feedback loop
• Only manual scores, no automated → quality is not continuously monitored
• All scores are identical → eval criteria too loose or rubric too vague
• Scores declining over time → model degradation or prompt drift

2e. Session and User Attribution

npx langfuse-cli api sessions list --limit 20

Verify:

• Sessions group related interactions (multi-turn conversations have one session ID)
• User IDs are attributed (not all anonymous)
• Session metadata is useful (page, feature, user segment)

2f. Dataset Health

npx langfuse-cli api datasets list

Evaluate:

• Datasets exist: Are there regression test datasets?
• Dataset freshness: When were items last added?
• Coverage: Do datasets cover all AI features or just one?
• Expected outputs: Do dataset items have expected outputs for automated comparison?

---

Phase 3: Live Verification

3a. Trigger AI Features via Playwright

For each AI feature identified in Phase 0c, use browser MCP tools to trigger it live.

Important: Apply the browser-anti-stall protocol — set 15-second timeouts, skip browser_wait_for on navigation, use browser_snapshot to detect ready state.

playwright:browser_navigate
{
 "url": "<APP_URL>"
}

Navigate to the feature, interact with it (fill form, click button, send message), and capture:

• The AI-generated response (via browser_snapshot)
• Console messages (via browser_console_messages) — look for errors
• Network requests (via browser_network_requests) — look for failed API calls

3b. Verify Trace Pipeline

After triggering each feature, wait 5-10 seconds, then verify the trace landed:

npx langfuse-cli api traces list --limit 5

Check:

• New trace appeared with correct name
• Trace has generations with model and token data
• Trace latency matches observed UX latency
• Input/output captured correctly

If a trace is missing after triggering a feature → pipeline break (critical finding).

3c. Cross-Check with Sentry

sentry:search_issues
{
 "organizationSlug": "<ORG_SLUG>",
 "projectSlug": "<PROJECT_SLUG>",
 "query": "is:unresolved ai OR llm OR openai OR anthropic OR langfuse OR completion OR embedding"
}

Check for:

• LLM timeout errors
• Rate limiting (429) errors
• Token limit exceeded errors
• Langfuse SDK errors (failed to send trace)
• JSON parse errors on LLM responses

3d. Cross-Check with Supabase (if AI results stored in DB)

If the project stores AI outputs in the database:

supabase:list_tables
{
 "project_id": "<PROJECT_ID>"
}

Find tables that store AI outputs and verify data landed:

supabase:execute_sql
{
 "project_id": "<PROJECT_ID>",
 "query": "SELECT id, created_at, <ai_output_column> FROM <table> ORDER BY created_at DESC LIMIT 5"
}

3e. Grounding / Hallucination Check

For features where the AI should reference source data (RAG, summarization, data extraction):

1. Get the source data from the database (Supabase execute_sql)
2. Trigger the AI feature via Playwright
3. Compare the AI output against the source data

Red flags:

• AI mentions facts not in the source data → hallucination
• AI omits critical facts from the source data → incomplete extraction
• AI contradicts the source data → grounding failure
• AI generates plausible but wrong numbers → numerical hallucination

---

Phase 4: Report

Generate a structured report with the following sections.

═══════════════════════════════════════════════════════
 LANGFUSE LLM QUALITY AUDIT REPORT
 Project: <PROJECT_NAME>
 Date: <DATE>
 Langfuse Host: <HOST_URL>
═══════════════════════════════════════════════════════

## 1. TRACE COVERAGE

| Feature | Traced? | Generations? | Input/Output? | Metadata? | Status |
|---------|---------|-------------|---------------|-----------|--------|
| ... | ... | ... | ... | ... | ✅/❌ |

Coverage: X/Y features traced (Z%)
Missing instrumentation: [list features with no traces]

## 2. PROMPT QUALITY

| Prompt | Version | Label | System Msg | Few-Shot | Guardrails | Variables | Score |
|--------|---------|-------|------------|----------|------------|-----------|-------|
| ... | ... | ... | ... | ... | ... | ... | A-F |

Hardcoded prompts found: [list files with inline prompts]
Recommendations: [specific improvements per prompt]

## 3. COST EFFICIENCY

| Feature | Model | Avg Tokens (in/out) | Avg Latency | Est. Cost/Call | Recommendation |
|---------|-------|---------------------|-------------|----------------|----------------|
| ... | ... | ... | ... | ... | ... |

Monthly estimate: $X (at current usage rate)
Savings opportunity: $Y (by implementing recommendations)

## 4. EVAL HEALTH

| Metric | Status | Details |
|------------------|-----------|----------------------------------|
| Automated evals | ✅/❌ | [count and types] |
| Manual reviews | ✅/❌ | [annotation queue status] |
| Score distribution| ✅/❌ | [healthy spread vs clustered] |
| Datasets | ✅/❌ | [count, freshness, coverage] |
| Regression tests | ✅/❌ | [dataset run frequency] |

## 5. PIPELINE INTEGRITY

| Step | Status | Evidence |
|-------------------------|--------|-------------------------------------|
| FE triggers AI feature | ✅/❌ | [Playwright observation] |
| API receives request | ✅/❌ | [network request captured] |
| LLM call executes | ✅/❌ | [trace generation exists] |
| Trace lands in Langfuse | ✅/❌ | [CLI verification] |
| Result stored in DB | ✅/❌ | [Supabase query result] |
| Result displayed in FE | ✅/❌ | [Playwright snapshot] |
| Eval score recorded | ✅/❌ | [score attached to trace] |

## 6. GROUNDING & HALLUCINATION

| Feature | Source Data | AI Output Match | Hallucinations | Score |
|---------|-------------|-----------------|----------------|-------|
| ... | ... | ... | ... | A-F |

## 7. SENTRY LLM ERRORS

| Issue | Error Type | Events | Impact | Fix Needed |
|-------|------------|--------|--------|------------|
| ... | ... | ... | ... | ... |

## 8. CRITICAL FINDINGS (Action Required)

P0 — Must fix immediately:
1. [finding with evidence]

P1 — Should fix this sprint:
1. [finding with evidence]

P2 — Improvement opportunity:
1. [finding with evidence]

## 9. RECOMMENDATIONS

| # | Category | Current State | Recommended State | Effort | Impact |
|---|----------|---------------|-------------------|--------|--------|
| 1 | ... | ... | ... | S/M/L | S/M/L |

## 10. PDCA IMPROVEMENT RESULTS

| Prompt | Baseline Score | Iter 1 Score | Iter 2 Score | Iter 3 Score | Final Score | Action Taken |
|--------|---------------|-------------|-------------|-------------|-------------|--------------|
| ... | ... | ... | ... | ... | ... | Promoted / Rolled back / Needs manual |

## Further reading

- [Improvement Details and more](references/details.md)