Istio validation debugger

istio-validation-debugger

Execution Mode — FULLY AUTONOMOUS

NEVER ask the user for confirmation. Fetch logs, interpret, propose and apply fixes immediately. The only user-facing handoff in the loop is triggering rollouts (owned by istio-iteration-loop). This skill never has user-facing handoffs.

Goal

Turn an Oracle or No-Op failure into a specific root cause with a specific fix, from log evidence — not guesswork. If you cannot map the pattern to a row below with confidence, investigate manually from the output.txt setup log + a re-read of setup.sh/solution.sh/grader.py before changing any file. (The live red-team / horizon-agentic-reviewer is DISABLED for this task — do not invoke it.)

Trigger

Oracle ≠ 1.0/passed; No-Op score > 0 (false positive); grader exception; feedback: null (setup crash); or you are about to edit setup/solution/grader after a validation failure. NOT for rollout failures — use istio-score-tuner.

Task Identity

Field	Value
Task UUID	<TASK_ID>
Slug / path	variation-istio-split-brain / tasks/variation-istio-split-brain/
Subscores	routing_restored (0.5) + mesh_governance (0.5), both binary
Venv	source /Users/mac/Documents/tasks/horizon_env/bin/activate
Grader version	v2 — includes the admission webhook, the bleater-env drift, the HPA/finalizer protections, and the two /etc/bleater decorrelating gates

v2 root-only files the grader reads: /opt/grader-scenario.json (keys vs_reconciler, ef_reconciler, env_reconciler), /opt/<grader-artefact>.txt, /opt/<grader-artefact>.txt. Agent-readable mirrors: /etc/bleater/<artefact>.txt, /etc/bleater/<artefact>.txt. If a /opt/grader-*.txt artefact is missing the grader SKIPS that gate (no deadweight) — so a "governance/routing always-1" with the value-gate silently skipped means the setup write failed; check output.txt.

Grader feedback is routing: <detail>; governance: <detail> — read both halves; each names the layer that fired.

Result shapes

score	passed	feedback	meaning
0	false	null	container/setup crash before grader ran → output.txt
0	false	non-null	both subscores failed (grader ran)
0.5	false	non-null	one passed, one failed
1.0	true	non-null	both passed

For No-Op, anything but score: 0 is a problem.

Step 1 — Fetch + read result

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate-logs -a oracle tasks/variation-istio-split-brain 2>&1
cat tasks/variation-istio-split-brain/.validation/*/result.json 2>/dev/null | python3 -m json.tool

Logs: tasks/variation-istio-split-brain/.validation/<build_id>/{output.txt,result.json}.

Step 2 — Map symptom → cause → fix

Branch A. feedback: null / score: 0 — setup crash (istio-specific causes first)

Pattern in output.txt	Root cause	Fix
error: unable to recognize ... no matches for kind "VirtualService" / EnvoyFilter etc.	The minimal Istio CRDs didn't install or didn't establish before the CR applies	Confirm the 6 install_open_crd calls run AND the kubectl wait --for=condition=established loop precedes the first mesh-object apply. The faults apply too early if the wait is missing.
timed out waiting for the condition on a crd	establish-wait too short or apiserver slow	The wait is --timeout=60s with `
k3s is not ready after 180 seconds	ENABLE_ISTIO_BLEATER=true in Dockerfile	Remove it entirely
Error from server (Forbidden) on kubectl get nodes	rbac yaml not copied / wrong path	Restore COPY data/ubuntu-user-rbac.yaml ...
failed to pull image for a drift/backend/webhook pod	external registry referenced	Use ${SIDECAR_IMAGE} (air-gapped)
setup aborts during the webhook/cert/HPA block	a command failed under set -e	The webhook block is wrapped in set +e ... set -e and the HPA apply ends in `
webhook pod CrashLoopBackOff / logs python3 unavailable; webhook inert	no python3 in ${SIDECAR_IMAGE}	Expected-tolerable: failurePolicy: Ignore means writes pass unmutated; the webhook is inert but its config still must be removed (Layer 6 still gates). Not a setup failure
setup.sh: line N: syntax error	bash error	bash -n setup.sh

Branch B. No-Op score > 0 — false positive

Subscore passed on No-Op	Why wrong	Fix
routing_restored=1	The faults didn't apply (CRD race) OR drift didn't deploy so the broken objects were never (re)written OR the route-policy gate silently SKIPPED (/opt/<grader-artefact>.txt missing)	Confirm the broken VS/EF/SE applied (Branch A CRD check), the reconcilers exist, and /opt/<grader-artefact>.txt was written (the /etc/bleater block ran)
mesh_governance=1	Drift didn't deploy, OR the broken RA/authz weren't injected, OR the VS already has <metadata-key>/<metadata-key> at setup time, OR the <metadata-key> gate silently SKIPPED (/opt/<grader-artefact>.txt missing), OR the broken VS already happens to satisfy a gate	Confirm setup sets the wrong VS metadata (<metadata-key>=experimental, no <metadata-key>/<metadata-key>), Service has NO <metadata-key>, attacker JWKS, blanket-deny authz, all FOUR drift deploys (incl. bleater-env) + the webhook deployed, and /opt/<grader-artefact>.txt was written

Branch C. Oracle 0.5 — routing_restored=0 (governance passed)

Read the routing: half of feedback:

Feedback fragment	Cause	Fix in solution.sh
edge EnvoyFilter ... still clamps route timeout	EnvoyFilter re-applied mid-window	Stop BOTH re-appliers before deleting it: the EF kube-system reconciler (sweep EF_RECONCILER_POOL) AND the canary envoy-patcher (delete the canary). Then kubectl delete envoyfilter bleater-edge-timeout -n bleater
routes to host ... with no backing Service (blackhole)	VS reconciler re-applied the broken VS	Stop the VS kube-system reconciler (sweep VS_RECONCILER_POOL) before re-applying the corrected VS pointing at bleater-timeline-service
delegates to namespace ... that does not exist	corrected VS still has the bleater-routes-ext delegate, OR the webhook re-mutated the VS, OR the ENV reconciler (bleater-env) re-applied the broken VS	Re-apply VS without the delegate route; delete the bleater-mesh-guard MutatingWebhookConfiguration FIRST; stop the ENV reconciler (sweep ENV_RECONCILER_POOL in bleater-env) AND the VS reconciler
mirrors traffic to ... blackhole sink	corrected VS still has the devnull mirror, OR webhook/ENV/VS reconciler re-broke it	Re-apply VS without mirror/mirrorPercentage; ensure webhook deleted + both VS re-appliers stopped
ServiceEntry ... hijacks in-cluster hostname	bleater-mesh-operator re-applied the SE	Stop bleater-mesh-operator (bleater) — delete its HPA bleater-mesh-operator-hpa first (or it bounces back) — before kubectl delete serviceentry bleater-timeline-resolution -n bleater
does not bind the platform route-policy token from /etc/bleater/<artefact>.txt to its route via a DestinationRule subset	[v4 routing-only multi-step gate] route isn't pinned to a subset, or no DestinationRule defines that subset with the token label	Read /etc/bleater/<artefact>.txt; create a DestinationRule on the timeline host with a subset labelled <anykey>=<token> AND set the VS primary route destination.subset to that subset name. If this is the ONLY routing failure and Oracle should be 1.0, solution.sh is missing the DR or the subset wiring (check Step 8/8b)

Branch D. Oracle 0.5 — mesh_governance=0 (routing passed)

Feedback fragment	Cause	Fix in solution.sh
VirtualService missing required label <metadata-key>=stable / annotation <metadata-key>=enforcing	corrected VS lacks the baseline metadata	Add label bleater.io/<metadata-key>: stable + annotation bleater.io/<metadata-key>: enforcing to the VS
does not anchor to the current platform install id from /etc/bleater/<artefact>.txt	[v2 governance-only gate] VS has no annotation whose value == the install id	Read /etc/bleater/<artefact>.txt; add an annotation with that value (solution uses bleater.io/<metadata-key>). If this is the only governance failure and Oracle should be 1.0, solution.sh dropped the annotation
timeline Service missing required annotation <metadata-key>=internal	[v2] Service not annotated	kubectl annotate service bleater-timeline-service -n bleater bleater.io/<metadata-key>=internal --overwrite
bleater-jwt ... does not validate against documented issuer/jwks or still trusts attacker JWKS	RA not reconciled	Apply RA with issuer=https://auth.devops.local, jwksUri=http://auth.devops.local/.well-known/jwks.json
AuthorizationPolicy ... is a blanket DENY	authz still deny-all	Delete bleater-edge-authz or set action ALLOW
active drift ... <name>@kube-system	a kube-system reconciler still running (the VS-pool one carries a finalizer — plain delete hangs)	Sweep BOTH pools; clear finalizers (kubectl patch ... -p '{"metadata":{"finalizers":[]}}') before delete, or scale to 0
active drift ... <name>@bleater-env	[v2] the ENV reconciler still running	Sweep ENV_RECONCILER_POOL in bleater-env; delete the bleater-side Role/RoleBinding + bleater-env SA. Needs ALLOWED_NAMESPACES to include bleater-env
active drift ... bleater-mesh-operator	local operator running, HPA bounces it back to 1	Delete HPA bleater-mesh-operator-hpa FIRST, then delete the deployment
cronjob mesh-config-refresher still active	refresher not stopped	delete or spec.suspend=true
bleater-timeline-canary mutation capability not fully neutralized	both sidecar AND runtime RBAC present	delete the canary deployment OR remove bleater-timeline-canary-runtime SA/Role/RoleBinding
MutatingWebhookConfiguration ... still intercepts mesh-object writes	[v2] the admission webhook config not removed	kubectl delete mutatingwebhookconfiguration bleater-mesh-guard (solution does this FIRST, before any VS/EF apply, so the fix isn't re-mutated). Needs the ubuntu-user-mesh-admission ClusterRole

Branch E. Oracle 0 non-null — both failed

Read both halves; follow Branch C and Branch D in parallel. If it stays 0 after fixing feedback, re-run No-Op to isolate a setup-side issue.

Branch F. Grader Python exception

json.JSONDecodeError/KeyError/TimeoutExpired — all guarded by kubectl_json/run_kubectl/.get. If one crept past, wrap it; treat failure as subscore=0. Not in the hard-constraint list — fix in place.

---

Step 3 — Pre-fix safety

1. Dockerfile + CRD invariants hold (guardian Step 0).
2. bash -n setup.sh && bash -n solution.sh && python3 -m py_compile grader.py.
3. If it touches subscore difficulty → diagnose with istio-score-tuner first.
4. Articulate the why in one sentence.

Fix type	Hand off to
setup.sh fault-injection bug	istio-task-guardian Step 3 (push)
solution.sh bug (Branch C/D)	istio-task-guardian Step 3
grader exception (Branch F)	istio-task-guardian Step 3
subscore difficulty	istio-score-tuner first
unmappable	manual investigation (read output.txt + setup.sh/solution.sh/grader.py); live red-team is disabled

Known non-issues (do NOT fix)

Pattern	Why
v1_feedback section FAILs	reviewer can't read heredoc baseline-config; dislikes pool-sweep oracle. Accepted
No-Op score: 0 with feedback	grader working correctly
governance fails because agents only audited bleater	the kube-system discovery axis working as intended
behavior_in_tests complaint (graded config not traffic)	correct by design — no istiod; grade shape

Verification cheat sheet

Hypothesis	Probe
"CRDs install + establish before CR apply"	grep -n "install_open_crd|--for=condition=established" setup.sh — establish loop must precede the kubectl apply -f of the broken objects
"solution deletes the webhook FIRST, before any VS/EF apply"	read solution.sh — delete mutatingwebhookconfiguration must precede the VS/EF deletes and the corrected VS apply (else the apply is re-mutated)
"solution stops EF's TWO re-appliers before deleting EF"	read solution.sh — EF reconciler sweep + canary delete must precede delete envoyfilter
"solution stops VS's TWO re-appliers"	read solution.sh — both VS reconciler pool sweep AND ENV_RECONCILER_POOL (bleater-env) sweep present
"solution sweeps all three pools"	grep -E "VS_RECONCILER_POOL|EF_RECONCILER_POOL|ENV_RECONCILER_POOL" solution.sh
"solution clears finalizers + deletes HPA before the operator"	`grep -nE "finalizers
"solution reflects the two /etc/bleater artefacts"	grep -nE "<artefact>.txt|<artefact>.txt|<metadata-key>|x-bleater-route-policy" solution.sh
"grader reads root-only mirrors, not /etc/bleater"	grep -nE "/opt/grader-(install-id|route-policy).txt|env_reconciler" grader.py
"wait_consistent unchanged"	grep -E "wait_consistent" grader.py — routing passes=5,gap=8,timeout=220; governance passes=5,gap=8,timeout=200
"no behavioral check crept into grader"	grep -iE "s_client|curl|openssl|http_code|connect" grader.py — must return nothing (config-shape only)
"no specifics leaked to task.yaml"	grep -iE "<metadata-key>|<metadata-key>|<metadata-key>|<metadata-key>|x-bleater-route-policy|install-id|route-policy|kube-system|bleater-env|reconciler|mesh-guard" task.yaml — must return nothing