agentsclimarketplace

Review and promote traces

Skill Goodeye-Labs/truesight-mcp-skills/skills/review-and-promote-traces

Agent skills for the Truesight MCP. Step-by-step workflow playbooks for scoring inputs, building live evaluations, error analysis, and the review loop. Works with Claude Code, Cursor, ChatGPT, VS Code, Windsurf, and any client that supports the agent skills standard.

Install
npx -y skills add Goodeye-Labs/truesight-mcp-skills --skill review-and-promote-traces

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 7 stars7 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Judge flagged trace outputs and promote judged items back to datasets. Use when an evaluation run requires human judgment or when review queue items need to be judged for promotion into the dataset.

SKILL.md

3.3 KB, as published. Nobody here has run it

Review and Promote Traces

Use this skill for the judgment and promotion loop after trace evaluation.

Interactive Q&A protocol (mandatory)

<HARD-GATE> BEFORE the first scoping question, search for a structured question tool (e.g., `AskUserQuestion` or similar interactive widget) and load it. Use that tool for EVERY scoping question. Fall back to plain-text lettered options ONLY if no such tool exists in the environment. </HARD-GATE>

If context is unclear, ask one question at a time using the structured question tool (loaded per the HARD-GATE above).

Example question structure:

What do you want to review first?
A) One specific run
B) Pending queue triage across runs

Rules:

  • Ask one question per message.
  • Use the structured question tool for every question. Structure each with a short header, 2-4 options with labels and descriptions, and place the recommended option first. Do not add "(Recommended)" or similar annotations to option labels.
  • Ask one follow-up when needed, then continue.

Workflow

  1. If needed, flag evaluated run:
    • Use flag_review_item with run_id.
  2. Retrieve review items:
    • Use list_review_items with pagination.
  3. Collect human judgments:
    • Present review items to the user and ask for judgment one item at a time when needed.
    • Capture judgment_value and optional notes for each item.
    • Use the structured question tool (loaded per the HARD-GATE above) to present options that match the live evaluation setup:
      • Binary example:
        • A) Pass
        • B) Fail
      • Categorical example:
        • A) <option 1>
        • B) <option 2>
        • C) <option 3>
      • Continuous example:
        • A) Enter numeric value (within configured range)
        • B) Skip this item for now
    • If valid options are unclear, look them up before asking:
      1. Inspect list_review_items payload for item result details and expected value hints.
      2. Use get_result(run_id) for run-level context and chain outputs.
      3. If evaluation id is available in run metadata, call get_evaluation(evaluation_id) and read config/judgment criteria to derive valid judgment options.
    • When options remain ambiguous after lookup, ask one clarification question using the structured question tool before proceeding.
  4. Submit judgments:
    • For each target item, call judge_review_item.
    • Pass the user-provided judgment_value and optional notes into judge_review_item.
    • Include notes when judgment context matters.
  5. Promote judged outputs:
    • Use add_reviewed_items_to_dataset with run_id.
  6. Report result:
    • number of items judged
    • promotion status and row counts
    • any skipped or blocked items

Queue-wide triage guidance

  • Group by run and status first.
  • Prioritize high-impact runs or oldest pending runs.
  • Keep an audit trail of judgment rationale in notes.

Scopes reference

  • list_review_items requires review:read
  • flag_review_item, judge_review_item, and add_reviewed_items_to_dataset require review:write

If a scope error occurs, ask the user to create a key with the missing scope in Truesight Settings.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.