Add llm evals

Skill ContextJet-ai/awesome-llm-observability/skills/add-llm-evals

50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

Install

npx -y skills add ContextJet-ai/awesome-llm-observability --skill add-llm-evals

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.

What its author says it does

Copied from the file, not written here

Use this when adding evaluation to an LLM/agent app - measuring output quality (correctness, faithfulness, relevance, safety) rather than just watching traces. Trigger on "add evals", "test my prompt", "is my RAG accurate", "catch regressions", "score outputs", or setting up an eval suite in CI. Covers offline (CI) and online (production LLM-as-a-judge) evaluation.

The file declares its own license as CC0-1.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

3.4 KB, as published. Nobody here has run it

Add evaluations to an LLM app

Observability tells you what happened; evaluation tells you whether it was good. Add both an offline suite (runs in CI on a fixed dataset) and, optionally, online scoring (grades production traffic).

Pick the eval type per metric

Two families, use both where relevant:

Reference-based (you have a ground-truth answer): exact/fuzzy match, semantic similarity, ROUGE/BLEU - cheap, deterministic. Good for classification, extraction, closed QA.
Reference-free / LLM-as-a-judge (open-ended output): a strong model scores the output against a rubric (faithfulness, relevance, coherence, safety). Scalable proxy for human judgment - but calibrate it (see caveats).

For RAG specifically, use the standard quartet: faithfulness, answer relevance, context precision, context recall (Ragas implements these).

Build the offline suite (do this first)

Curate a dataset - 20-100 representative inputs with expected outputs or rubrics. Pull real cases from production traces (this is where observability + eval connect). Version it in the repo.
Choose a framework - promptfoo (YAML, great for prompt/RAG + CI), DeepEval (pytest-style, 40+ metrics), or Ragas (RAG metrics). Pick one; don't hand-roll.
Define pass criteria - per-metric thresholds (e.g. faithfulness ≥ 0.8). Fail the build if a threshold regresses.
Wire into CI - run on every PR that touches prompts/models/retrieval. Store scores so you can see trends, not just pass/fail.

See references/frameworks.md for a minimal promptfoo config and a DeepEval test example.

Add online evaluation (optional, higher value)

Sample production traffic and score it with LLM-as-a-judge (most observability platforms - Langfuse, Phoenix, Opik, Braintrust - run these on live traces). Alert when a quality score drops. This catches drift the offline suite can't (real inputs shift over time).

Calibrate LLM-as-a-judge (don't skip)

Anchor to humans: label ~30 examples yourself, then check the judge agrees. Iterate the rubric until agreement is high.
Watch known biases: position bias, verbosity bias, self-preference. Randomize order; keep rubrics concrete.
Use a strong judge model and a low temperature; a weak judge produces noisy scores.

Verify

Offline suite runs green in CI and fails when you intentionally break a prompt.
Scores are stored/trended, not just printed.
(If online) a dashboard shows quality over time with an alert threshold.

Anti-patterns

Only "vibe-checking" outputs manually - doesn't scale, doesn't catch regressions.
A dataset of toy inputs that don't resemble production.
Trusting an uncalibrated LLM judge as ground truth.
Evals that never fail the build (then they're decoration, not a gate).

Research grounding

LLM-as-a-judge: Zheng et al. 2023 (MT-Bench/Chatbot Arena). Reference-free hallucination detection: SelfCheckGPT (Manakul et al. 2023). See this repo's README → Research & Benchmarks.