agentsclimarketplace

Rca annotator

Skill redhat-et/rhdp-rca-plugin/skills/rca-annotator

Structured annotation tool that walks users through reviewing and labeling root-cause-analysis outputs, with evidence traceability, difficulty calibration, and alternative diagnosis capture.From its SKILL.md

Install
npx -y skills add redhat-et/rhdp-rca-plugin --skill rca-annotator

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 15 stars15 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

8.6 KB, as published. Nobody here has run it

RCA Annotator

A structured annotation tool that presents the root-cause-analysis agent's diagnosis to the user and guides them through labeling it — capturing whether the diagnosis is correct, evidence quality, difficulty, and alternative hypotheses.

root-cause-analysis (Agent)rca-annotator (Annotation tool)
PurposeDiagnose failures for usersCapture human-labeled ground-truth data
ReadsLogs, Splunk, GitHub (live)Step 1/3/4/5 output files (offline)
OutputHuman-readable diagnosisStructured annotation.json

Use when: a root-cause-analysis run is complete and you want to annotate its output as correct, incorrect, or partially correct — for evaluation, benchmarking, or dataset building.

Do NOT use to perform initial RCA (use root-cause-analysis).

Prerequisites

Verify root-cause-analysis has been completed.

  • JUMPBOX_URI (optional) — SSH connection string (e.g. "user@host -p 2222"). If unset, uses local .analysis/ only.
  • SSH keys configured in ~/.ssh/config if using jumpbox; ssh and rsync installed.
  • Required files in .analysis/<job_id>/:
    • step5_summary.json — Agent's final diagnosis (primary input)
    • step1_job_context.json — Job metadata, failed tasks
    • step3_correlation.json — Timeline with AAP + Splunk events
    • step4_github_fetch_history.json — Configuration and code context

If missing, run root-cause-analysis skill first.

Workflow

  1. Download from jumpbox (if JUMPBOX_URI set) or verify local files
  2. Read step5_summary.json — present the agent's diagnosis to the user
  3. Walk through annotation questions interactively — the user labels each section
  4. Write annotation.json with the user's labels
  5. Upload to jumpbox (if JUMPBOX_URI set)

Step 0: Download Analysis Files

cd skills/rca-annotator
python scripts/cli.py download --job-id <job_id>

Downloads from jumpbox /usr/local/mlflow/<job_id>/ to local .analysis/<job_id>/. If JUMPBOX_URI unset, validates local files only. Errors on missing remote directory, missing required files, or connection failure.


Step 1: Read Agent Diagnosis

Read step5_summary.json and present the agent's diagnosis clearly to the user:

  • Root cause category and summary
  • Confidence level
  • Key evidence cited
  • Difficulty score (if present)
  • Recommendations
  • Alternative diagnoses (if any)

This is the starting point for annotation. The user is reviewing the agent's work.


Step 2: Interactive Annotation

Walk through each question below with the user. Present the relevant section from step5_summary.json before asking each question. Wait for the user's response before continuing.

1. Root Cause Category

Present the agent's category and summary. Ask:

Is the root cause category correct? (e.g. configuration, infrastructure, credential — or should it be something else?)

Valid categories: configuration | infrastructure | application_bug | dependency | network | resource | cloud_api | credential | secrets | unknown

2. Summary Accuracy

Present the agent's summary sentence. Ask:

Is the summary accurate and specific? (Does it clearly describe what failed and why?)

3. Evidence

Present the evidence items the agent cited. Ask:

Is any evidence missing or wrong? (Any key log lines, config values, or Splunk events that were overlooked or incorrectly cited?)

If the user wants to cross-check, read step1/step3/step4 and compare against what the agent cited. This is reference material for validation — not a re-analysis.

Evidence traceability format (for any new or corrected evidence items the user provides):

{
  "source": "step1 | step3 | step4",
  "source_file": ".analysis/<job_id>/step1_job_context.json",
  "json_path": "failed_tasks[0].duration",
  "exact_value": 917.565567,
  "exact_quote": "optional — literal text for code/config",
  "line_number": 5,
  "github_path": "owner/repo:path/to/file.yml:line",
  "message": "The relevant log line or config snippet.",
  "confidence": "high | medium | low",
  "is_root_cause": true
}

4. Difficulty Rating

Present the agent's difficulty score (or estimate one from the evidence). Present the calibration rubric to help the user score:

CriterionPoints
Requires cross-source correlation (AAP + Splunk + GitHub)+3
Requires understanding code behavior+2
Error message is generic or misleading+2
Requires variable precedence/override knowledge+1
Requires domain knowledge (K8s, Ansible, cloud APIs)+1
Multiple plausible alternatives exist+1
Timing dependencies are critical+1

Mapping: 0–3 = easy, 4–6 = medium, 7–10 = hard.

Ask:

Is the difficulty rating appropriate? (Score of X / 10 — too easy, too hard, or about right? Use the rubric above if helpful.)

5. Alternative Diagnoses

Present any alternative diagnoses the agent identified. Ask:

Any alternative diagnoses to add or correct? (Other plausible-but-incorrect hypotheses worth capturing?)

Alternative diagnosis format:

{
  "category": "infrastructure",
  "summary": "A plausible but wrong diagnosis.",
  "why_wrong": "Why the evidence does not support this.",
  "plausibility": "high | medium | low",
  "supporting_evidence": ["long timeout", "destroy action"],
  "contradicting_evidence": ["test environment", "auth retry pattern"]
}

plausibility: high = shares many characteristics | medium = some evidence | low = superficial similarity


Step 3: Write Annotation

After all questions are answered, verify before writing:

  • Root cause category confirmed or corrected
  • Exactly one evidence item has is_root_cause: true
  • All evidence has traceability (source_file, json_path, exact_value/quote)
  • Difficulty score calculated with justification
  • Alternative diagnoses have plausibility levels

Write annotation.json to .analysis/<job_id>/.


Step 4: Upload Annotation

cd skills/rca-annotator
python scripts/cli.py upload --job-id <job_id>

Uploads .analysis/<job_id>/annotation.json to jumpbox if JUMPBOX_URI set. Local copy always preserved. If JUMPBOX_URI unset, file remains local only.


Output Format

Save to .analysis/<job_id>/annotation.json:

{
  "job_id": "1234567",
  "annotated_at": "2026-03-19T12:05:00Z",

  "category_correct": true,
  "category_comment": "Confirmed — matches the auth retry pattern.",

  "root_cause": {
    "category": "configuration | infrastructure | application_bug | dependency | network | resource | cloud_api | credential | secrets | unknown",
    "summary": "One sentence describing what failed and why.",
    "confidence": "high | medium | low"
  },

  "summary_accurate": true,
  "summary_comment": "Clear and specific.",

  "evidence": [
    {
      "source": "step1 | step3 | step4",
      "source_file": ".analysis/<job_id>/step1_job_context.json",
      "json_path": "failed_tasks[0].duration",
      "exact_value": 917.565567,
      "exact_quote": "optional — literal text for code/config",
      "line_number": 5,
      "github_path": "owner/repo:path/to/file.yml:line",
      "message": "The relevant log line or config snippet.",
      "confidence": "high | medium | low",
      "is_root_cause": true
    }
  ],

  "evidence_feedback": "Missing the kubeconfig 404 from step4 github_fetches.",

  "difficulty": "easy | medium | hard",
  "difficulty_score": 5,
  "difficulty_justification": "Requires correlating task code (+2) with missing configs and interpreting generic MODULE FAILURE (+2). Total: 5.",
  "difficulty_appropriate": false,
  "difficulty_comment": "Should be hard (8/10) — requires deep variable precedence knowledge.",

  "recommendations": [
    {
      "priority": "high | medium | low",
      "action": "What should be done to fix it.",
      "file": "path/to/file.yml"
    }
  ],

  "contributing_factors": [
    "Factor that made the failure more likely or harder to diagnose."
  ],

  "alternative_diagnoses": [
    {
      "category": "infrastructure",
      "summary": "A plausible but wrong diagnosis.",
      "why_wrong": "Why the evidence does not support this.",
      "plausibility": "high | medium | low",
      "supporting_evidence": ["long timeout", "destroy action"],
      "contradicting_evidence": ["test environment", "auth retry pattern"]
    }
  ]
}

Keep looking

Skills are one crate of 611,178. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.