Mtls validation debugger

mtls-validation-debugger

Execution Mode — FULLY AUTONOMOUS

NEVER ask the user for confirmation. Fetch logs, interpret results, propose and apply fixes immediately. The only user-facing handoff in the loop is triggering rollouts in the Horizon UI (owned by mtls-iteration-loop Step 4c). This skill never has user-facing handoffs.

Goal

Turn an Oracle or No-Op failure into a specific root cause with a specific fix, using only the log evidence — not guesswork. The point of this skill is to prevent blind edits. If you cannot map the observed log pattern to one of the rows below with confidence, escalate to horizon-agentic-reviewer for a live probe before changing any file.

Trigger

Use this skill when:

• Oracle returned anything other than score: 1.0, passed: true
• No-Op returned a score > 0 (false positive — the grader passed on a broken setup)
• The grader crashed with a Python exception
• result.json shows feedback: null (container/setup crash before grader ran)
• You are about to edit setup.sh, solution.sh, or grader.py in response to a validation failure

Do not use this skill for rollout failures (those are scoring/variance issues — use mtls-score-tuner). Oracle and No-Op are validation; rollouts are evaluation.

Task Identity

Field	Value
Task UUID	<TASK_ID>
Task slug	variation-fanout-pipeline-break
Local path	tasks/variation-fanout-pipeline-break/
Subscores	mtls_handshake (0.5) + trust_governance (0.5), both binary
Horizon root	/Users/mac/Documents/tasks
Venv	source /Users/mac/Documents/tasks/horizon_env/bin/activate

How to read a validation result (read once, internalise)

Oracle runs setup.sh → solution.sh → grader.py. No-Op runs setup.sh → grader.py (no solution). The four possible result shapes:

score	passed	feedback	Meaning	Where to look
0	false	null	Container or setup crashed before the grader could run	output.txt for the crash trace
0	false	non-null string	Grader ran end-to-end, both subscores failed	grader feedback for the specific check that fired
0.5	false	non-null string	One subscore passed, the other failed	feedback names which subscore is 1 vs 0
1.0	true	non-null string	Both subscores passed	nothing to debug

For No-Op, anything other than score: 0 is a problem — the grader should never pass without solution.sh having run. A No-Op pass means the grader is too lax or setup.sh didn't actually break the world.

---

Step 1 — Fetch the logs

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

# Oracle (setup + solution + grader)
horizon tasks validate-logs -a oracle tasks/variation-fanout-pipeline-break 2>&1

# No-Op (setup + grader, no solution) — only when iteration-loop says to run it
horizon tasks validate-logs -a noop tasks/variation-fanout-pipeline-break 2>&1

Logs persist under:

• tasks/variation-fanout-pipeline-break/.validation/<build_id>/output.txt
• tasks/variation-fanout-pipeline-break/.validation/<build_id>/result.json

Step 2 — Read result.json first

cat tasks/variation-fanout-pipeline-break/.validation/*/result.json 2>/dev/null | python3 -m json.tool

The four-row table above tells you which symptom branch to follow in Step 3. Do not skip to output.txt without checking result.json first — the result shape narrows the search dramatically.

Step 3 — Map symptom → cause → fix

Branch A. feedback: null / score: 0 — setup crash

Setup never finished, so the grader never ran. Search output.txt for the first error line:

Pattern in output.txt	Root cause	Fix
k3s is not ready after 180 seconds	ENABLE_ISTIO_BLEATER=true is in the Dockerfile, causing 17+ pod force-deletes on startup	Remove ENABLE_ISTIO_BLEATER from the Dockerfile entirely
Error from server (Forbidden) on kubectl get nodes	data/ubuntu-user-rbac.yaml not copied into image, or COPY path wrong	Restore COPY data/ubuntu-user-rbac.yaml /mcp_server/Nebula/infra/k8s/rbac/ubuntu-user-rbac.yaml
Error from server (Forbidden) on kube-system resources	ALLOWED_NAMESPACES env var missing	Add ENV ALLOWED_NAMESPACES="kube-system" to Dockerfile
failed to pull image ... bleater-profile-cache-sync	Strategy A deployment is referencing an external registry image	Use ${SIDECAR_IMAGE} (the in-cluster reference); cluster is air-gapped
Error: failed to pull image (any other image)	Base image or tag wrong	Verify FROM line in Dockerfile
kubectl: command not found	Wrong base image	Verify FROM line in Dockerfile
setup.sh: line N: syntax error	Bash syntax error	Run bash -n setup.sh locally — fix that line
Setup stalls / hits hosted timeout with no error	k3s init slow OR a kubectl wait is hanging	First verify ENABLE_ISTIO_BLEATER absent; then add --timeout= flags to any unbounded kubectl wait

Branch B. No-Op returned score > 0 — false positive

The grader passed without solution.sh having run. Either setup.sh did not break the world, or the grader is too lax.

Subscore that passed on No-Op	Why it shouldn't have	Fix
mtls_handshake = 1 on No-Op	Drift didn't bite — either bleater-profile-cache-sync or node-telemetry-batcher failed to deploy, OR the wait_consistent window is shorter than both their cadences	kubectl get deploy bleater-profile-cache-sync -n bleater (must exist, replicas=1) AND kubectl get cronjob node-telemetry-batcher -n kube-system (must exist, not suspended); also confirm grader.py wait_consistent window is 240s (handshake) — anything shorter than ~120s lets a lucky agent slip through
trust_governance = 1 on No-Op	One or more drift controllers didn't deploy. Check setup.sh fault-injection section is reaching all four targets	grep -E "cert-reaper|cache-sync|canary|hpa-pinner|kube-system.*python" /tmp/noop_log.txt — every name must appear in the setup output

If setup.sh appears to claim it deployed everything but the cluster is missing pieces, add set -ex to setup.sh temporarily, push, and re-run No-Op to get verbose trace. The fault-injection section often fails silently on || true paths.

Branch C. Oracle score: 0.5 — mtls_handshake failed (governance passed)

Solution.sh stopped the drift controllers but the TLS handshake never recovered. Search for handshake-relevant log lines:

grep -i "handshake\|cert\|ssl\|tls\|verify" /tmp/oracle_log.txt | head -30

Common causes, ordered by likelihood:

Cause	Why	Fix in solution.sh
Pods not bounced after Secret patch	New cert is on disk but the running process holds the old one; kubelet's ~60s Secret-projection delay keeps the new mount hidden	Add kubectl delete pod -l app=<profile-service-pod-label> -n bleater --wait=false AND kubectl delete pod -l app=<bleat-service-pod-label> -n bleater --wait=false after the Secret patch. solution.sh lines 201-202 do this.
Wrong CA Subject CN on regenerated CA	grader.py line 174 checks cn=bleater-ca; agent regenerated with a different CN	The CA must have Subject CN = bleater-ca — openssl req -x509 -subj "/CN=bleater-ca" .... The baseline config /home/ubuntu/profile-mtls-baseline-config.txt documents this.
Wrong SAN on regenerated leaf cert	grader.py uses openssl s_client -verify_hostname; agent's SAN doesn't include the headless FQDN	The leaf cert SAN must include bleater-profile-service-mtls.bleater.svc.cluster.local (not bleater-profile-service...). Check the subjectAltName in the openssl/cfssl config.
Trust bundle ConfigMap not repatched	Solution regenerated CA but forgot to update bleater-trust-bundle ConfigMap key ca.crt	kubectl patch configmap bleater-trust-bundle -n bleater --type merge -p ... with the new PEM-encoded CA
Secret not patched	Solution regenerated the leaf cert but forgot to write it to bleater-profile-mtls (keys tls.crt, tls.key)	kubectl patch secret bleater-profile-mtls -n bleater --type merge ... with the new material
bleater-profile-cache-sync still running	Strategy A deployment is still re-patching after solution.sh fixes the Secret	kubectl scale deployment bleater-profile-cache-sync -n bleater --replicas=0 before cert regeneration
node-telemetry-batcher CronJob still active	kube-system CronJob fires every 60s mid-window	kubectl delete cronjob node-telemetry-batcher -n kube-system
Grader connects before pods are ready	Pods are still restarting when the grader probe fires	The wait_consistent retry loop handles this most of the time; if persistent, add kubectl wait --for=condition=Ready pod -l app=... -n bleater --timeout=90s after the delete

Branch D. Oracle score: 0.5 — trust_governance failed (handshake passed)

Cert handshake recovered but at least one drift controller is still re-breaking trust during the grader window.

grep -iE "cronjob|cert-reaper|cache-sync|hpa|canary|drift|governance" /tmp/oracle_log.txt | head -30

Common causes, ordered by likelihood:

Cause	Why	Fix in solution.sh
bleater-profile-cert-rotator CronJob not deleted/suspended	grader.py Layer 1 — must be absent OR spec.suspend=true	kubectl delete cronjob bleater-profile-cert-rotator -n bleater (and its -n bleater SA/Role/RoleBinding)
cluster-trust-syncer Deployment still running in kube-system	grader.py Layer 2 — must be replicas=0 AND readyReplicas=0	kubectl scale deployment cluster-trust-syncer -n kube-system --replicas=0
bleater-mesh-validator Deployment still running in kube-system	grader.py Layer 2 — same	kubectl scale deployment bleater-mesh-validator -n kube-system --replicas=0
HPA pinner still active on bleater-cert-reaper	Solution scaled bleater-cert-reaper to 0 but bleater-cert-reaper-pinner HPA immediately restored replicas=1	kubectl delete hpa bleater-cert-reaper-pinner -n bleater before scaling. The [11:04] incident-notes line warns about this.
bleater-cert-reaper Deployment still running in bleater	grader.py Layer 2 — same as the kube-system entries (replicas=0, readyReplicas=0)	After deleting the HPA: kubectl scale deployment bleater-cert-reaper -n bleater --replicas=0
bleater-profile-canary mutation capability still intact	grader.py Layer 3 — needs sidecar OR -runtime RBAC removed (either is sufficient)	Easiest: kubectl delete deployment bleater-profile-canary -n bleater + delete the canary-runtime SA/Role/RoleBinding
Grader's wait_consistent window too short	A controller briefly dies and re-spawns inside the 40s window	Do not increase past passes=5, gap=8s, timeout=200s — at the ceiling, this is the wrong fix. Investigate which controller is respawning.

Branch E. Oracle score: 0 with non-null feedback — both subscores failed

Both subscores failed cleanly (the grader ran end-to-end). Most likely setup.sh did its job correctly but solution.sh is broken in a way that misses both dimensions. Read the feedback for each subscore independently — it usually names the specific check that fired. Then follow Branch C for mtls_handshake and Branch D for trust_governance in parallel.

If Oracle returns score: 0 repeatedly even after fixing what the feedback says, suspect a setup-side issue contaminating the solution run — re-run No-Op to isolate.

Branch F. Grader crashed with a Python exception

grep -A 5 "Traceback\|Exception\|Error" /tmp/oracle_log.txt | head -40

Exception	Likely cause	Fix in grader.py
subprocess.CalledProcessError	A kubectl invocation inside the grader returned non-zero	Add check=False to the subprocess.run and handle the error path explicitly
json.JSONDecodeError	kubectl ... -o json returned empty (resource not found) or non-JSON	Wrap json.loads() in try/except, treat parse failure as subscore=0
KeyError on a dict access	grader assumed a key exists that's missing in some cluster states	Use .get(key, default)
subprocess.TimeoutExpired	A kubectl wait is hanging	Add a finite timeout to every kubectl wait inside the grader; treat timeout as subscore=0

These are grader bugs, not task-design issues — fix them in place. They are not in the hard-constraint list.

---

Step 4 — Pre-fix safety checklist

Before applying any fix from Step 3, confirm:

1. Dockerfile invariants still hold — see mtls-task-guardian Step 0
2. Syntax is clean — bash -n setup.sh && bash -n solution.sh && python3 -m py_compile grader.py
3. The proposed change is in mtls-score-tuner's safe-lever table if it touches scoring difficulty
4. You can articulate the why — "this fixes X because Y" — in one sentence

Then hand off:

If your fix is	Hand off to
setup.sh fault-injection bug	mtls-task-guardian Step 3 (push) directly
solution.sh bug (Branch C/D)	mtls-task-guardian Step 3 directly — solution.sh is not a tuning surface
grader.py exception (Branch F)	mtls-task-guardian Step 3 directly
Anything affecting subscore difficulty	mtls-score-tuner for diagnosis first — do not edit grader without diagnosis
Anything you cannot map to a row above	horizon-agentic-reviewer to verify on a live container

---

Known non-issue patterns (do NOT fix these)

Pattern	Why it isn't a bug
Quality check 18/20 (or 15/20 on a noisy day)	Four checks (v1_feedback, behavior_in_task_documentation, behavior_in_tests, sometimes task_clarity) are LLM hallucinations against the heredoc baseline config. Permanent and acceptable — see mtls-task-guardian Step 2
No-Op returns score: 0 with detailed feedback	This is the grader working correctly — feedback names the missing dimensions
bleater-profile-cache-sync not discovered by agents in the first batch	Strategy A is intentionally hidden — not in baseline config, not named in the prompt. Discovery cost is the design
trust_governance failing on rollouts because agents only audited bleater namespace	This is the kube-system discovery axis working as intended — do not "fix" it by hinting at kube-system in the prompt

---

Verification cheat sheet

Hypothesis	Probe
"Setup.sh deployed every drift source the grader expects"	grep -E "cert-rotator|trust-syncer|mesh-validator|cert-reaper|profile-canary|cache-sync|telemetry-batcher|cert-reaper-pinner" /tmp/oracle_log.txt — all eight names should appear (5 governance-graded + cache-sync + telemetry-batcher + HPA pinner)
"Solution.sh runs the scale-down before the cert regen"	Read solution.sh; the kubectl scale ... --replicas=0 lines must appear above the cert regeneration block
"HPA delete happens before cert-reaper scale"	Read solution.sh; kubectl delete hpa bleater-cert-reaper-pinner must appear above kubectl scale deployment bleater-cert-reaper
"Pods bounced after Secret patch via kubectl delete pod"	Read solution.sh; kubectl delete pod -l app=<profile-label> AND kubectl delete pod -l app=<bleat-label> must appear after the Secret apply (NOT rollout restart — the actual solution.sh uses delete pod)
"Grader catches regenerated cert with wrong CN/SAN"	grep -E "cn=bleater-ca|verify_hostname" grader.py — these MUST appear; they catch agents who regenerate the CA with the wrong identity. Their absence would be a regression.
"No baseline-preserve cert check has crept in"	grep -iE "NotAfter|NotBefore|Issuer|Serial" grader.py — must return nothing (those are v6 dead-weight patterns)
"Wait_consistent params haven't drifted past ceiling"	grep -E "wait_consistent" grader.py — handshake exactly passes=5, gap=15, timeout=240; governance exactly passes=5, gap=8, timeout=200. Different values = drift.
"Setup.sh wipes events at the end"	grep -n "kubectl delete events" setup.sh — must appear before the hide-grader block (so agents can't reverse-engineer faults from event history)