Llm observability and evals
Skill wanghong5233/agent-engineering-kit/cursor/.cursor/skills/llm-observability-and-evals
Design observability and eval contracts for production LLM/Agent systems. Use when adding ask/retrieval/tool/model flows, debugging quality regressions, defining trace/log/metric fields, building golden sets, or proving an Agent feature works. Do NOT use for cost-only work.From its SKILL.md
npx -y skills add wanghong5233/agent-engineering-kit --skill llm-observability-and-evalsAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
3.4 KB, ~1.0k tokens by cl100k_base, as published. Nobody here has run it
LLM Observability & Evals
一句话准则
Agent 上线不是"能回答"就算完成，而是每次决策都能回放、每次质量变化都能被 eval 捕捉。日志是行为合同，eval 是质量合同。
最小上线合同
任何 LLM/Agent 路径上线前至少具备：
| 类别 | 必备字段 / 产物 |
|---|---|
| Trace | trace_id, conversation_id, attempt_id, job_id |
| Decision | route_reason, index_mode, tool_choice, model_route, policy_version |
| Execution | retrieval/tool/model 每段 latency_ms, status, error_class |
| Quality | golden set、pass threshold、失败样例归档 |
| Cost | input_tokens, output_tokens, model, provider, feature |
| UX state | accepted, running, failed, timed_out, completed |
缺一类时，先补 contract，再谈优化。
Observability 设计流程
- 画链路：用户输入 → route → retrieval/tool → model → persistence → UI。
- 标决策点：每个
if/route/fallback/retry都要有reason。 - 标分段耗时：routing / retrieval / tool / generation / postprocess 分开计。
- 统一字段名：同一概念只允许一个 schema 名。
- 采样策略：高频成功路径可采样，失败路径必须全量。
- 隐私审计：prompt 可摘要化，secret/token/cookie 不落日志。
Eval 设计流程
| 阶段 | 做什么 |
|---|---|
| Golden Set | 从真实 bug、用户问题、边界样例沉淀 20+ 条 |
| Metrics | 分类/抽取用 exact/schema；摘要/问答用 rubric 或 LLM-as-judge |
| Baseline | 记录当前生产 prompt/model/retrieval 的分数 |
| Gate | prompt、retrieval、model route 改动必须跑 eval |
| Drift | 定期重跑，发现模型更新或数据变化导致的退化 |
Agent 专属 Eval 维度
| 维度 | 问题 |
|---|---|
| Intent Routing | 是否按用户意图选择 RAG / direct context / tool |
| Retrieval Quality | 是否召回正确文档、正确片段、足够证据 |
| Tool Use | 是否选择正确工具、传参合法、处理错误 |
| Answer Grounding | 回答是否引用 evidence，而不是编造 |
| State Durability | 失败时用户输入和 attempt 状态是否可见 |
| Recovery UX | 失败是否可重试、错误是否可解释 |
反模式
| 反模式 | 风险 | 改法 |
|---|---|---|
| 只有最后答案，没有中间决策 | 无法解释为什么检索/调用工具 | 记录 decision + execution |
| 只靠人工点点看 | 回归不可重复 | golden set 进 CI |
| 日志字段各服务自定义 | 查询和 join 失败 | schema 进文档与测试 |
| 质量指标只看 thumbs up | 样本稀疏且滞后 | 线上反馈 + 离线 eval 双轨 |
| 失败只返回 500 | 用户和工程都无法恢复 | 结构化 error_class + retryable |
输出格式
## Trace Contract
## Metrics
## Eval Set
## Pass Criteria
## Dashboards / Alerts
## Gaps Before Launch
链接
- 上游来源与改写说明：
references/upstream.md - 成本优化：
llm-cost-optimizer - Prompt 版本治理可从本 skill 拆出
prompt-governance
What ships with it: 1 file
1.2 KB alongside SKILL.md
references/
- upstream.md1.2 KB