agentsclimarketplace
Sign in

Observe ai production

Skill hiteshbandhu/skills-i-use/skills/ai-engineer-talks/observe-ai-production

Observes and improves production LLM apps with Arize Phoenix—tracing, layered agent evals, prompt learning loops, and PM/engineering eval pipelines. Use when the user mentions Arize, Phoenix, OpenInference, agent router evals, prompt optimization from traces, or shipping AI with production observability.From its SKILL.md

Install
npx -y skills add hiteshbandhu/skills-i-use --skill observe-ai-production

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
  • runs commandsInstructs the agent to run 3 commands, including `cp -r skills/observe-ai-production ~/.claude/skills/` and 2 more.

SKILL.md

2.0 KB, 440 tokens by cl100k_base, as published. Nobody here has run it

Observe AI production (Arize)

Action playbook from six Arize @ AI Engineer talks. Do not summarize talks — pick a workflow and execute it.

Supporting files: workflows.md · source-index.md

Optional: ./skill-outputs/observe-ai-production/

Step 0 — Pick workflow

What is the user trying to do?
├─ Layered agent evals (router, skills, convergence)     → agent-evals
├─ Hands-on Phoenix agent eval workshop                  → phoenix-workshop
├─ Prompt learning from production traces                → prompt-learning
├─ PM-facing eval frameworks and release gates           → pm-evals
└─ Org-scale eval pipelines (CI, versioning)             → eval-pipelines

See workflows.md. Related: run-llm-evals for cross-vendor eval theory.

Install

cp -r skills/observe-ai-production ~/.claude/skills/
cp -r skills/observe-ai-production ~/.cursor/skills/
cp -r skills/observe-ai-production ~/.codex/skills/

Source: playlists/arize-ai-engineer/.

Cross-cutting rules

RuleSource
Eval router decisions, not only final answers[src-001 @ 0:08:21]
Trace every tool/LLM call for agent eval substrate[src-002 @ 0:06:18]
Calibrate LLM judges on human labels before automation[src-002 @ 0:89:37]
Close loop: observability → dataset → prompt patch[src-003 @ 0:20:00]
Version datasets/scorers like production code[src-005 @ 0:10:00]

Output

Name workflow; save artifacts to ./skill-outputs/observe-ai-production/ when requested; do not auto-commit.

What ships with it: 3 files

4.1 KB alongside SKILL.md

Gives 0 of the 12 instructions most evals benchmarks skills give in 440 tokens

Counted across 499 of the 513 authors here whose files we hold, read 2026-09-06

  • Spawn with-skill and baseline runs in the same turnin 31 of 499, across 24 files
  • Keep SKILL.md under 500 linesin 31 of 499, across 24 files
  • Draft assertions while test runs are in progressin 31 of 499, across 24 files
  • Compare against the baseline after changesin 31 of 499, across 13 files
  • Define evals before codingin 26 of 499, across 17 files
  • Run evals frequently during developmentin 25 of 499, across 16 files
  • Keep evals fastin 24 of 499, across 15 files
  • Version evals with codein 24 of 499, across 15 files
  • Generate the eval viewer before evaluating outputs yourselfin 24 of 499, across 17 files
  • Generate an eval report after runsin 24 of 499, across 15 files
  • Track pass@k metrics over timein 22 of 499, across 14 files
  • Save a baseline before making changesin 21 of 499, across 9 files

Said here and by no other author read

  • Name the chosen workflow and what you are producing
  • Save artifacts when the user wants files
  • Pick the matching workflow and execute it
  • Evaluate router decisions, not only final answers
  • Trace every tool and LLM call
  • Close the loop from observability to prompt patch

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.