Maddy validation debugger

maddy-validation-debugger

Execution Mode — FULLY AUTONOMOUS

NEVER ask the user for confirmation. Fetch logs, interpret results, propose and apply fixes immediately. The only user-facing handoff in the loop is triggering rollouts in the Horizon UI (owned by maddy-iteration-loop). This skill never has user-facing handoffs.

Goal

Turn an Oracle or No-Op failure into a specific root cause with a specific fix, using only the log evidence — not guesswork. If you cannot map the observed log pattern to one of the rows below with confidence, escalate to horizon-agentic-reviewer for a live probe before changing any file.

Trigger

Use this skill when:

• Oracle returned anything other than score: 1.0, passed: true
• No-Op returned a score > 0 (false positive — grader passed on a broken setup)
• The grader crashed with a Python exception
• result.json shows feedback: null (container/setup crash before grader ran)
• You are about to edit setup.sh, solution.sh, or grader.py in response to a validation failure

Do not use this skill for rollout failures (use maddy-score-tuner). Oracle/No-Op are validation; rollouts are evaluation.

Task Identity

Field	Value
Task UUID	<TASK_ID>
Task slug	maddy-delivery-split-brain
Local path	tasks/maddy-delivery-split-brain/
Subscores	delivery_consistency (0.5) + observability_governance (0.5), both binary
Horizon root	/Users/mac/Documents/tasks
Venv	source /Users/mac/Documents/tasks/horizon_env/bin/activate

How to read a validation result (read once, internalise)

Oracle runs setup.sh → solution.sh → grader.py. No-Op runs setup.sh → grader.py (no solution). The four result shapes:

score	passed	feedback	Meaning	Where to look
0	false	null	Container/setup crashed before grader ran	output.txt for the crash trace
0	false	non-null	Grader ran end-to-end, both subscores failed	grader feedback for the specific check that fired
0.5	false	non-null	One subscore passed, the other failed	feedback names which is 1 vs 0
1.0	true	non-null	Both passed	nothing to debug

For No-Op, anything other than score: 0 is a problem — the grader should never pass without solution.sh having run.

The grader feedback string is delivery_consistency: <detail>; observability_governance: <detail> — read both halves; each names the exact layer that fired.

---

Step 1 — Fetch the logs

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate-logs -a oracle tasks/maddy-delivery-split-brain 2>&1
# No-Op only when iteration-loop says to run it:
horizon tasks validate-logs -a noop tasks/maddy-delivery-split-brain 2>&1

Logs persist under tasks/maddy-delivery-split-brain/.validation/<build_id>/{output.txt,result.json}.

Step 2 — Read result.json first

cat tasks/maddy-delivery-split-brain/.validation/*/result.json 2>/dev/null | python3 -m json.tool

The four-row table tells you which branch to follow. Do not skip to output.txt without checking result.json first.

Step 3 — Map symptom → cause → fix

Branch A. feedback: null / score: 0 — setup crash

Pattern in output.txt	Root cause	Fix
k3s is not ready after 180 seconds	ENABLE_ISTIO_BLEATER=true in Dockerfile	Remove it entirely
Error from server (Forbidden) on kubectl get nodes	data/ubuntu-user-rbac.yaml not copied / wrong path	Restore the COPY data/ubuntu-user-rbac.yaml ... line
Setup hangs at Waiting for bleater-env namespace	bleater-env namespace never appears in this image	The base image must provision bleater-env; if it doesn't, gate the ENV_RECONCILER block on a bounded wait + `
failed to pull image for a drift/maddy pod	A pod references an external registry	Use ${SIDECAR_IMAGE} (derived from a live bleater deployment); cluster is air-gapped
setup.sh: line N: syntax error	Bash error	bash -n setup.sh, fix that line
Setup stalls to hosted timeout with no error	a kubectl rollout status / kubectl wait hanging	Confirm ENABLE_ISTIO_BLEATER absent; add --timeout= to any unbounded wait

NOTE: setup.sh waits for bleater-env with until kubectl get namespace bleater-env ...; do sleep 2; done (no timeout). If the base image does not create bleater-env, this is an infinite hang → setup-timeout crash. Verify the namespace exists on the live container (Step 4b of guardian). If it doesn't, that loop needs a bound.

Branch B. No-Op returned score > 0 — false positive

Subscore that passed on No-Op	Why it shouldn't have	Fix
delivery_consistency = 1 on No-Op	The split-brain didn't form — maddy-legacy didn't deploy, OR no app=maddy pod mounts maddy-config-legacy	kubectl get deploy maddy-legacy -n bleater (must exist, replicas≥1) and confirm its pod template mounts maddy-config-legacy
observability_governance = 1 on No-Op	Drift controllers didn't deploy, OR the GitOps source isn't stale, OR the per-pod-rule check is too lax	Confirm all three drift deployments exist (kube-system/bleater-env/bleater-canary), maddy-gitops-source says replicas: 1/mode: direct, and the maddy template has NO POD_NAME env at setup time

If setup claims it deployed everything but the cluster is missing pieces, add set -ex to setup.sh temporarily, push, re-run No-Op for a verbose trace. The fault-injection section can fail silently on || true paths.

Branch C. Oracle score: 0.5 — delivery_consistency failed (governance passed)

solution.sh stopped the drift and fixed observability, but the converged-delivery state didn't hold. The feedback string names the layer:

Feedback fragment	Root cause	Fix in solution.sh
pod ... still mounts legacy config maddy-config-legacy	A drift controller re-created the legacy replica during the window	solution.sh must stop all three drift sources BEFORE deleting maddy-legacy. Confirm it discovers + deletes: kube-system reconciler (annotation), bleater-env reconciler (annotation), canary config-pinner. A missed source re-spawns the legacy pod.
legacy Deployment maddy-legacy still running	maddy-legacy not deleted, or re-created by drift	kubectl delete deployment maddy-legacy -n bleater AFTER stopping drift
maddy rollout not converged (spec=.., updated=.., ready=..)	maddy not scaled to 2, or pods not Ready in time	Scale maddy to 2 and kubectl rollout status deployment/maddy -n bleater --timeout=180s before the grader probes
only N ready maddy pods on the smarthost config	Fewer than 2 Ready pods mount maddy-config	Ensure the converged maddy Deployment mounts maddy-config and reaches 2 Ready
maddy-config is not the smarthost relay config	The smarthost ConfigMap got clobbered	Do not edit maddy-config; it should already be smarthost. If a drift source rewrites it, that's a setup bug

Branch D. Oracle score: 0.5 — observability_governance failed (consistency passed)

Feedback fragment	Root cause	Fix in solution.sh
maddy container does not expose pod identity via the downward API	No POD_NAME env with fieldRef metadata.name on the converged maddy Deployment	Add env: [{name: POD_NAME, valueFrom: {fieldRef: {fieldPath: metadata.name}}}] to the maddy container
maddy-gitops-source still declares direct-delivery routing / does not declare replicas: 2	GitOps ConfigMap not reconciled	kubectl apply the maddy-gitops-source CM with values.yaml containing replicas: 2 and relay.mode: smarthost
no per-pod maddy delivery/divergence rule found	The rule ConfigMap is missing OR lacks a real PromQL by (pod) expression	Create a ConfigMap whose data has a - record:/- alert:/expr: rule that mentions maddy, a signal word (rate/success/delivery/divergence), AND a per-pod dimension by (pod). Bare keywords in comments do NOT pass.
active drift would re-introduce the skew: <name>@kube-system	kube-system reconciler still has replicas>0	Discover by annotation bleater.io/purpose=maddy rollout convergence, delete/scale to 0
active drift in bleater-env ...@bleater-env	bleater-env reconciler still running — OR agent could not reach bleater-env	If solution.sh can't even get -n bleater-env, this is the ALLOWED_NAMESPACES structural risk → fix Dockerfile (guardian Step 0), not solution.sh
<canary> mutation capability not fully neutralized	config-pinner sidecar AND <canary>-runtime RBAC both still present	Delete the canary Deployment OR remove the -runtime SA/Role/RoleBinding (either suffices)

Branch E. Oracle score: 0 with non-null feedback — both subscores failed

Both ran and failed cleanly. Read each half of the feedback independently and follow Branch C for delivery_consistency and Branch D for observability_governance in parallel. If Oracle stays 0 after fixing what feedback says, re-run No-Op to isolate a setup-side contamination.

Branch F. Grader crashed with a Python exception

grep -A 5 "Traceback\|Exception\|Error" /tmp/oracle_log.txt | head -40

Exception	Likely cause	Fix in grader.py
json.JSONDecodeError	kubectl ... -o json returned empty/non-JSON	Already guarded by kubectl_json; if a raw run_kubectl path crept in, wrap it
KeyError	assumed a dict key present in all cluster states	Use .get(key, default)
subprocess.TimeoutExpired	a kubectl call hung	run_kubectl has a finite timeout; check no unbounded shell pipeline was added
FileNotFoundError /opt/grader-scenario.json	scenario file not written by setup	<grader-check-fn> already falls back to defaults — if it's crashing, the try/except was removed; restore it

These are grader bugs — fix in place; they are not in the hard-constraint list.

---

Step 4 — Pre-fix safety checklist

Before applying any fix:

1. Dockerfile + namespace invariants still hold — see maddy-task-guardian Step 0 (especially bleater-env reachability)
2. Syntax clean — bash -n setup.sh && bash -n solution.sh && python3 -m py_compile grader.py
3. The change is in maddy-score-tuner's safe-lever table if it touches scoring difficulty
4. You can articulate the why in one sentence

Then hand off:

If your fix is	Hand off to
setup.sh fault-injection bug	maddy-task-guardian Step 3 (push) directly
solution.sh bug (Branch C/D)	maddy-task-guardian Step 3 directly — solution.sh is not a tuning surface
grader.py exception (Branch F)	maddy-task-guardian Step 3 directly
Anything affecting subscore difficulty	maddy-score-tuner for diagnosis first
Anything you cannot map to a row above	horizon-agentic-reviewer for a live probe

---

Known non-issue patterns (do NOT fix these)

Pattern	Why it isn't a bug
Quality check v1_feedback section returns FAIL	LLM reviewer can't read the heredoc baseline config / incident notes; objects to annotation-based discovery. Permanent, accepted — see maddy-task-guardian Step 2
No-Op returns score: 0 with detailed feedback	The grader working correctly — feedback names the missing dimensions
observability_governance failing on rollouts because agents only audited bleater	This is the cross-namespace (kube-system + bleater-env) discovery axis working as intended
Agents miss the bleater-env reconciler specifically	Intended — it has an innocuous non-maddy name; only the annotation links it. Discovery cost is the design

---

Verification cheat sheet

Hypothesis	Probe
"Setup deployed all three drift sources"	`grep -E "RECONCILER=
"solution.sh stops drift BEFORE deleting maddy-legacy"	Read solution.sh — the annotation/config-pinner discovery+delete block must appear above kubectl delete deployment maddy-legacy
"solution.sh adds POD_NAME + gitops + per-pod rule"	`grep -E "fieldPath: metadata.name
"Per-pod-rule gate hasn't been loosened"	`grep -E "by (pod)
"wait_consistent hasn't drifted past ceiling"	grep -E "wait_consistent" grader.py — consistency passes=5, gap=8, timeout=220; governance passes=5, gap=8, timeout=200
"bleater-env wait is bounded (no infinite hang)"	grep -n "Waiting for bleater-env" setup.sh then read the loop — if it's an unbounded until, confirm the namespace truly exists on the live image