Collect user feedback
Skill ContextJet-ai/awesome-llm-observability/skills/collect-user-feedback
Use this to capture user feedback on LLM outputs (thumbs up/down, edits, corrections, implicit signals) and feed it back into observability and evals. Trigger on "add thumbs up/down", "collect feedback on responses", "how do I know if users like the answers", "improve from real usage", "human feedback loop". Turn real user signal into your best source of eval data.From its SKILL.md
npx -y skills add ContextJet-ai/awesome-llm-observability --skill collect-user-feedbackAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
What its file declares
Copied from the file, not written here
The file declares its own license as CC0-1.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.
SKILL.md
3.5 KB, 743 tokens by cl100k_base, as published. Nobody here has run it
Collect user feedback
Your users are running the best eval you have, for free, every day. Capturing their reactions and attaching them to the trace turns production into a continuous source of labeled data. Most teams either skip this or collect it and never use it.
Capture both explicit and implicit signals
Explicit (ask directly):
- Thumbs up/down on a response (the simplest, highest-signal control).
- A short reason on thumbs-down (dropdown: wrong / unhelpful / unsafe / other).
- Accept / edit / reject on a suggested output (great for copilot-style apps).
Implicit (infer from behavior, no extra UI):
- Did the user retry, rephrase, or immediately give up? (dissatisfaction).
- Did they copy/use the output, or continue the task? (satisfaction).
- Did a downstream action succeed (the code ran, the ticket resolved)?
Implicit signals are noisier but you get them on 100% of traffic instead of the small % who click thumbs.
Wire it to the trace (this is the key step)
Attach every feedback signal to the trace/span it is about (by trace ID), with the user, timestamp, and the reason. Most observability platforms (Langfuse, Phoenix, Opik, LangSmith) have a feedback/scores API for exactly this. Without the trace link, feedback is a number with no context; with it, a thumbs-down is a fully debuggable example.
Use it (do not just collect it)
- Monitor feedback rate + thumbs-down rate over time; alert on spikes (see
set-up-drift-alerts). - Triage thumbs-down traces into candidate test cases (see
trace-based-testing) and your eval set (build-eval-dataset). - Correlate feedback with your automated eval scores to check your judge actually agrees with humans (calibration).
- Close the loop: the corrections/edits users make are gold-standard reference outputs, use them.
Handle it responsibly
- PII / privacy: feedback text can contain sensitive data, redact before it hits a third-party backend (
redact-pii-for-tracing). - Bias: thumbs are a biased sample (angry and delighted users click most). Do not treat thumbs-up rate as ground-truth quality; use it as signal, validate with evals.
Verify
- A thumbs-down in the UI shows up attached to the right trace in your backend.
- Thumbs-down traces are being triaged into the eval/regression set, not just counted.
- A dashboard shows feedback trends with an alert on a drop.
Anti-patterns
- Collecting feedback that never links to a trace (a number you cannot act on).
- Collecting it and never feeding it back into evals or fixes.
- Treating thumbs-up rate as objective quality (selection bias).
- Logging raw feedback with PII to a SaaS backend without redaction.
Grounding
Human feedback as the signal for LLM quality underpins RLHF and preference modeling (Ouyang et al., InstructGPT, arXiv:2203.02155) and human-aligned evaluation (Zheng et al. 2023, arXiv:2306.05685). Here it feeds the lighter-weight eval + observability loop (trace-based-testing, build-eval-dataset).
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.
Gives 0 of the 12 instructions most evals benchmarks skills give in 743 tokens
Counted across 499 of the 513 authors here whose files we hold, read 2026-09-06
- Spawn with-skill and baseline runs in the same turnin 31 of 499, across 24 files
- Keep SKILL.md under 500 linesin 31 of 499, across 24 files
- Draft assertions while test runs are in progressin 31 of 499, across 24 files
- Compare against the baseline after changesin 31 of 499, across 13 files
- Define evals before codingin 26 of 499, across 17 files
- Run evals frequently during developmentin 25 of 499, across 16 files
- Keep evals fastin 24 of 499, across 15 files
- Version evals with codein 24 of 499, across 15 files
- Generate the eval viewer before evaluating outputs yourselfin 24 of 499, across 17 files
- Generate an eval report after runsin 24 of 499, across 15 files
- Track pass@k metrics over timein 22 of 499, across 14 files
- Save a baseline before making changesin 21 of 499, across 9 files
Said here and by no other author read
- Capture explicit and implicit user feedback signals
- Attach every feedback signal to its trace ID
- Record user, timestamp, and reason with each signal
- Monitor feedback and thumbs-down rates over time
- Alert on feedback rate spikes
- Triage thumbs-down traces into candidate test cases
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.