Rca annotator
Structured annotation tool that walks users through reviewing and labeling root-cause-analysis outputs, with evidence traceability, difficulty calibration, and alternative diagnosis capture.From its SKILL.md
npx -y skills add redhat-et/rhdp-rca-plugin --skill rca-annotatorAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 15 stars15 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
8.6 KB, as published. Nobody here has run it
RCA Annotator
A structured annotation tool that presents the root-cause-analysis agent's diagnosis to the user and guides them through labeling it — capturing whether the diagnosis is correct, evidence quality, difficulty, and alternative hypotheses.
root-cause-analysis (Agent) | rca-annotator (Annotation tool) | |
|---|---|---|
| Purpose | Diagnose failures for users | Capture human-labeled ground-truth data |
| Reads | Logs, Splunk, GitHub (live) | Step 1/3/4/5 output files (offline) |
| Output | Human-readable diagnosis | Structured annotation.json |
Use when: a root-cause-analysis run is complete and you want to annotate its output as correct, incorrect, or partially correct — for evaluation, benchmarking, or dataset building.
Do NOT use to perform initial RCA (use root-cause-analysis).
Prerequisites
Verify root-cause-analysis has been completed.
JUMPBOX_URI(optional) — SSH connection string (e.g."user@host -p 2222"). If unset, uses local.analysis/only.- SSH keys configured in
~/.ssh/configif using jumpbox;sshandrsyncinstalled. - Required files in
.analysis/<job_id>/:step5_summary.json— Agent's final diagnosis (primary input)step1_job_context.json— Job metadata, failed tasksstep3_correlation.json— Timeline with AAP + Splunk eventsstep4_github_fetch_history.json— Configuration and code context
If missing, run root-cause-analysis skill first.
Workflow
- Download from jumpbox (if
JUMPBOX_URIset) or verify local files - Read
step5_summary.json— present the agent's diagnosis to the user - Walk through annotation questions interactively — the user labels each section
- Write
annotation.jsonwith the user's labels - Upload to jumpbox (if
JUMPBOX_URIset)
Step 0: Download Analysis Files
cd skills/rca-annotator
python scripts/cli.py download --job-id <job_id>
Downloads from jumpbox /usr/local/mlflow/<job_id>/ to local .analysis/<job_id>/. If JUMPBOX_URI unset, validates local files only. Errors on missing remote directory, missing required files, or connection failure.
Step 1: Read Agent Diagnosis
Read step5_summary.json and present the agent's diagnosis clearly to the user:
- Root cause category and summary
- Confidence level
- Key evidence cited
- Difficulty score (if present)
- Recommendations
- Alternative diagnoses (if any)
This is the starting point for annotation. The user is reviewing the agent's work.
Step 2: Interactive Annotation
Walk through each question below with the user. Present the relevant section from step5_summary.json before asking each question. Wait for the user's response before continuing.
1. Root Cause Category
Present the agent's category and summary. Ask:
Is the root cause category correct? (e.g.
configuration,infrastructure,credential— or should it be something else?)
Valid categories: configuration | infrastructure | application_bug | dependency | network | resource | cloud_api | credential | secrets | unknown
2. Summary Accuracy
Present the agent's summary sentence. Ask:
Is the summary accurate and specific? (Does it clearly describe what failed and why?)
3. Evidence
Present the evidence items the agent cited. Ask:
Is any evidence missing or wrong? (Any key log lines, config values, or Splunk events that were overlooked or incorrectly cited?)
If the user wants to cross-check, read step1/step3/step4 and compare against what the agent cited. This is reference material for validation — not a re-analysis.
Evidence traceability format (for any new or corrected evidence items the user provides):
{
"source": "step1 | step3 | step4",
"source_file": ".analysis/<job_id>/step1_job_context.json",
"json_path": "failed_tasks[0].duration",
"exact_value": 917.565567,
"exact_quote": "optional — literal text for code/config",
"line_number": 5,
"github_path": "owner/repo:path/to/file.yml:line",
"message": "The relevant log line or config snippet.",
"confidence": "high | medium | low",
"is_root_cause": true
}
4. Difficulty Rating
Present the agent's difficulty score (or estimate one from the evidence). Present the calibration rubric to help the user score:
| Criterion | Points |
|---|---|
| Requires cross-source correlation (AAP + Splunk + GitHub) | +3 |
| Requires understanding code behavior | +2 |
| Error message is generic or misleading | +2 |
| Requires variable precedence/override knowledge | +1 |
| Requires domain knowledge (K8s, Ansible, cloud APIs) | +1 |
| Multiple plausible alternatives exist | +1 |
| Timing dependencies are critical | +1 |
Mapping: 0–3 = easy, 4–6 = medium, 7–10 = hard.
Ask:
Is the difficulty rating appropriate? (Score of X / 10 — too easy, too hard, or about right? Use the rubric above if helpful.)
5. Alternative Diagnoses
Present any alternative diagnoses the agent identified. Ask:
Any alternative diagnoses to add or correct? (Other plausible-but-incorrect hypotheses worth capturing?)
Alternative diagnosis format:
{
"category": "infrastructure",
"summary": "A plausible but wrong diagnosis.",
"why_wrong": "Why the evidence does not support this.",
"plausibility": "high | medium | low",
"supporting_evidence": ["long timeout", "destroy action"],
"contradicting_evidence": ["test environment", "auth retry pattern"]
}
plausibility: high = shares many characteristics | medium = some evidence | low = superficial similarity
Step 3: Write Annotation
After all questions are answered, verify before writing:
- Root cause category confirmed or corrected
- Exactly one evidence item has
is_root_cause: true - All evidence has traceability (source_file, json_path, exact_value/quote)
- Difficulty score calculated with justification
- Alternative diagnoses have plausibility levels
Write annotation.json to .analysis/<job_id>/.
Step 4: Upload Annotation
cd skills/rca-annotator
python scripts/cli.py upload --job-id <job_id>
Uploads .analysis/<job_id>/annotation.json to jumpbox if JUMPBOX_URI set. Local copy always preserved. If JUMPBOX_URI unset, file remains local only.
Output Format
Save to .analysis/<job_id>/annotation.json:
{
"job_id": "1234567",
"annotated_at": "2026-03-19T12:05:00Z",
"category_correct": true,
"category_comment": "Confirmed — matches the auth retry pattern.",
"root_cause": {
"category": "configuration | infrastructure | application_bug | dependency | network | resource | cloud_api | credential | secrets | unknown",
"summary": "One sentence describing what failed and why.",
"confidence": "high | medium | low"
},
"summary_accurate": true,
"summary_comment": "Clear and specific.",
"evidence": [
{
"source": "step1 | step3 | step4",
"source_file": ".analysis/<job_id>/step1_job_context.json",
"json_path": "failed_tasks[0].duration",
"exact_value": 917.565567,
"exact_quote": "optional — literal text for code/config",
"line_number": 5,
"github_path": "owner/repo:path/to/file.yml:line",
"message": "The relevant log line or config snippet.",
"confidence": "high | medium | low",
"is_root_cause": true
}
],
"evidence_feedback": "Missing the kubeconfig 404 from step4 github_fetches.",
"difficulty": "easy | medium | hard",
"difficulty_score": 5,
"difficulty_justification": "Requires correlating task code (+2) with missing configs and interpreting generic MODULE FAILURE (+2). Total: 5.",
"difficulty_appropriate": false,
"difficulty_comment": "Should be hard (8/10) — requires deep variable precedence knowledge.",
"recommendations": [
{
"priority": "high | medium | low",
"action": "What should be done to fix it.",
"file": "path/to/file.yml"
}
],
"contributing_factors": [
"Factor that made the failure more likely or harder to diagnose."
],
"alternative_diagnoses": [
{
"category": "infrastructure",
"summary": "A plausible but wrong diagnosis.",
"why_wrong": "Why the evidence does not support this.",
"plausibility": "high | medium | low",
"supporting_evidence": ["long timeout", "destroy action"],
"contradicting_evidence": ["test environment", "auth retry pattern"]
}
]
}