Fanout cve validation debugger
Skill omonuj/claude-horizon-skills/skills/fanout-cve-rollout/fanout-cve-validation-debugger
Claude Code Agent Skills for building, red-teaming and tuning agentic RL evaluation environments — a four-skill pattern (guardian, validation-debugger, score-tuner, iteration-loop) plus a 24-point adversarial reviewer.
npx -y skills add omonuj/claude-horizon-skills --skill fanout-cve-validation-debuggerAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
3 things to look at
- 13 days oldThe repository was created 13 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Fetches and interprets validation logs (No-Op and Oracle) for the fanout-cve-rollout task (<TASK_ID>). Maps each of the 5 reward-shaped subscores' failures to a root cause and the smallest safe fix. Use after a validation fails before editing anything.
SKILL.md
6.0 KB, as published. Nobody here has run it
fanout-cve-validation-debugger
Execution Mode — FULLY AUTONOMOUS
NEVER ask for confirmation. Fetch logs, interpret, fix immediately. No user-facing handoffs. Can't map a pattern → escalate to horizon-agentic-reviewer.
Trigger
Oracle ≠ 1.0/passed; No-Op score > 0; grader exception; feedback: null (setup crash); or before editing setup/solution/grader. NOT for rollout-gradient issues — use fanout-cve-score-tuner.
Task Identity
| Field | Value |
|---|---|
| UUID | <TASK_ID> |
| Slug / path | fanout-cve-rollout / tasks/fanout-cve-rollout/ |
| Subscores (0.2 each) | cve_image_patched, configmap_reverted, pods_rolled_live_env, gitops_committed, argocd_synced |
| Venv | source /Users/mac/Documents/tasks/horizon_env/bin/activate |
Grader feedback is name=0/1 (detail) for each subscore, joined by ; , plus a partial-fix HINT. Root-only file the grader trusts: /opt/grader-cve.json (patched_version, vulnerable_versions). Oracle = 1.0 only when all five are 1.
Step 1 — Fetch + read
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate-logs -a oracle tasks/fanout-cve-rollout 2>&1
cat tasks/fanout-cve-rollout/.validation/*/result.json 2>/dev/null | python3 -m json.tool
Step 2 — Map symptom → cause → fix
Branch A. feedback: null / score: 0 — setup crash
output.txt pattern | Cause | Fix |
|---|---|---|
k3s is not ready after 180 seconds | ENABLE_ISTIO_BLEATER=true | remove it |
Forbidden on kubectl get nodes | rbac not copied | restore COPY data/ubuntu-user-rbac.yaml |
failed to pull image for the fanout pod | external registry referenced | use ${SIDECAR_IMAGE} — the fanout pods MUST be runnable (the grader execs them) |
| fanout rollout never converges in setup | image unrunnable / resource limits | confirm ${SIDECAR_IMAGE} + sane limits |
setup.sh: line N: syntax error | bash error | bash -n setup.sh |
Branch B. No-Op score > 0 — false positive
| Subscore | Why wrong on No-Op | Fix |
|---|---|---|
configmap_reverted=1 | setup didn't patch the CM to peer | confirm the kubectl patch configmap ... peer runs AFTER the rollout |
pods_rolled_live_env=1 | pods snapshotted peer not full, OR setup never created the drift | setup must create the CM at full, roll the Deployment, THEN patch CM→peer (so live env=full while CM=peer) |
cve_image_patched=1 | the Deployment shipped a clean version | setup must label the template bleater.io/fanout-version=1.4.2 (vulnerable) |
gitops_committed=1 / argocd_synced=1 | helm-values already patched / argocd already Synced | setup must seed fanout-helm-values with the vulnerable version and argocd-bleater-app status: OutOfSync |
Branch C. Oracle partial (< 1.0) — one or more subscores failed
Read each name=0 (detail) and fix the matching solution.sh step:
| Failed subscore | Cause | Fix in solution.sh |
|---|---|---|
cve_image_patched | template version label still vulnerable | patch the Deployment template label bleater.io/fanout-version to the patched version (this also rolls the pods) |
configmap_reverted | CM still peer | kubectl patch configmap fanout-tls-config ... full |
pods_rolled_live_env + HINT fired | pods not rolled, or rolled while CM was still peer | revert the CM to full FIRST, then bump the version (rolls pods → they snapshot full); ensure kubectl rollout status converges before grading |
pods_rolled_live_env (no hint, version mismatch) | a stale pod from the old ReplicaSet survived | ensure the rollout fully converged (updated==ready==replicas) |
gitops_committed | helm-values not updated | apply fanout-helm-values with the patched fanoutVersion + amqpTlsValidationMode: full |
argocd_synced | argocd status not Synced | kubectl patch configmap argocd-bleater-app ... {"status":"Synced"} |
Branch D. Grader exception
The exec helper retries and tolerates failure (returns None → subscore 0 with a clear message). JSON/kubectl access is guarded. If a new crash appears, wrap it; treat failure as subscore=0.
Common ordering bug (the #1 oracle failure)
If the solution bumps the version (rolls pods) BEFORE reverting the ConfigMap, the new pods snapshot peer → pods_rolled_live_env fails on live env peer. Always revert the ConfigMap to full first, then roll.
Step 3 — Pre-fix safety
- Dockerfile invariants hold.
bash -n setup.sh && bash -n solution.sh && py_compile grader.py.- Subscore-design change → diagnose with
fanout-cve-score-tunerfirst. - Articulate the why.
| Fix type | Hand off |
|---|---|
| setup fault-injection / solution ordering / grader exception | fanout-cve-task-guardian Step 3 (push) |
| subscore independence / partial-credit design | fanout-cve-score-tuner |
| unmappable | horizon-agentic-reviewer |
Known non-issues (do NOT fix)
| Pattern | Why |
|---|---|
v1_feedback FAIL ("ConfigMap schemas missing") | reviewer can't read the heredoc ConfigMap data. Accepted noise |
| Oracle partial during a fix iteration | expected — the gradient is dense; read which subscore is 0 |
Verification cheat sheet
| Hypothesis | Probe |
|---|---|
| "solution reverts CM before rolling" | read solution.sh — the fanout-tls-config ... full patch precedes the version bump |
| "grader execs live pod env" | `grep -E "exec |
| "grader trusts root-only CVE truth" | `grep -E "/opt/grader-cve.json |
| "5 independent subscores, 0.2 each" | `grep -E "weights |