Fanout cve validation debugger

fanout-cve-validation-debugger

Execution Mode — FULLY AUTONOMOUS

NEVER ask for confirmation. Fetch logs, interpret, fix immediately. No user-facing handoffs. Can't map a pattern → escalate to horizon-agentic-reviewer.

Trigger

Oracle ≠ 1.0/passed; No-Op score > 0; grader exception; feedback: null (setup crash); or before editing setup/solution/grader. NOT for rollout-gradient issues — use fanout-cve-score-tuner.

Task Identity

Field	Value
UUID	<TASK_ID>
Slug / path	fanout-cve-rollout / tasks/fanout-cve-rollout/
Subscores (0.2 each)	cve_image_patched, configmap_reverted, pods_rolled_live_env, gitops_committed, argocd_synced
Venv	source /Users/mac/Documents/tasks/horizon_env/bin/activate

Grader feedback is name=0/1 (detail) for each subscore, joined by ; , plus a partial-fix HINT. Root-only file the grader trusts: /opt/grader-cve.json (patched_version, vulnerable_versions). Oracle = 1.0 only when all five are 1.

Step 1 — Fetch + read

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate-logs -a oracle tasks/fanout-cve-rollout 2>&1
cat tasks/fanout-cve-rollout/.validation/*/result.json 2>/dev/null | python3 -m json.tool

Step 2 — Map symptom → cause → fix

Branch A. feedback: null / score: 0 — setup crash

output.txt pattern	Cause	Fix
k3s is not ready after 180 seconds	ENABLE_ISTIO_BLEATER=true	remove it
Forbidden on kubectl get nodes	rbac not copied	restore COPY data/ubuntu-user-rbac.yaml
failed to pull image for the fanout pod	external registry referenced	use ${SIDECAR_IMAGE} — the fanout pods MUST be runnable (the grader execs them)
fanout rollout never converges in setup	image unrunnable / resource limits	confirm ${SIDECAR_IMAGE} + sane limits
setup.sh: line N: syntax error	bash error	bash -n setup.sh

Branch B. No-Op score > 0 — false positive

Subscore	Why wrong on No-Op	Fix
configmap_reverted=1	setup didn't patch the CM to peer	confirm the kubectl patch configmap ... peer runs AFTER the rollout
pods_rolled_live_env=1	pods snapshotted peer not full, OR setup never created the drift	setup must create the CM at full, roll the Deployment, THEN patch CM→peer (so live env=full while CM=peer)
cve_image_patched=1	the Deployment shipped a clean version	setup must label the template bleater.io/fanout-version=1.4.2 (vulnerable)
gitops_committed=1 / argocd_synced=1	helm-values already patched / argocd already Synced	setup must seed fanout-helm-values with the vulnerable version and argocd-bleater-app status: OutOfSync

Branch C. Oracle partial (< 1.0) — one or more subscores failed

Read each name=0 (detail) and fix the matching solution.sh step:

Failed subscore	Cause	Fix in solution.sh
cve_image_patched	template version label still vulnerable	patch the Deployment template label bleater.io/fanout-version to the patched version (this also rolls the pods)
configmap_reverted	CM still peer	kubectl patch configmap fanout-tls-config ... full
pods_rolled_live_env + HINT fired	pods not rolled, or rolled while CM was still peer	revert the CM to full FIRST, then bump the version (rolls pods → they snapshot full); ensure kubectl rollout status converges before grading
pods_rolled_live_env (no hint, version mismatch)	a stale pod from the old ReplicaSet survived	ensure the rollout fully converged (updated==ready==replicas)
gitops_committed	helm-values not updated	apply fanout-helm-values with the patched fanoutVersion + amqpTlsValidationMode: full
argocd_synced	argocd status not Synced	kubectl patch configmap argocd-bleater-app ... {"status":"Synced"}

Branch D. Grader exception

The exec helper retries and tolerates failure (returns None → subscore 0 with a clear message). JSON/kubectl access is guarded. If a new crash appears, wrap it; treat failure as subscore=0.

Common ordering bug (the #1 oracle failure)

If the solution bumps the version (rolls pods) BEFORE reverting the ConfigMap, the new pods snapshot peer → pods_rolled_live_env fails on live env peer. Always revert the ConfigMap to full first, then roll.

Step 3 — Pre-fix safety

1. Dockerfile invariants hold.
2. bash -n setup.sh && bash -n solution.sh && py_compile grader.py.
3. Subscore-design change → diagnose with fanout-cve-score-tuner first.
4. Articulate the why.

Fix type	Hand off
setup fault-injection / solution ordering / grader exception	fanout-cve-task-guardian Step 3 (push)
subscore independence / partial-credit design	fanout-cve-score-tuner
unmappable	horizon-agentic-reviewer

Known non-issues (do NOT fix)

Pattern	Why
v1_feedback FAIL ("ConfigMap schemas missing")	reviewer can't read the heredoc ConfigMap data. Accepted noise
Oracle partial during a fix iteration	expected — the gradient is dense; read which subscore is 0

Verification cheat sheet

Hypothesis	Probe
"solution reverts CM before rolling"	read solution.sh — the fanout-tls-config ... full patch precedes the version bump
"grader execs live pod env"	`grep -E "exec
"grader trusts root-only CVE truth"	`grep -E "/opt/grader-cve.json
"5 independent subscores, 0.2 each"	`grep -E "weights