Oncall validation debugger

oncall-validation-debugger

Execution Mode — FULLY AUTONOMOUS

NEVER ask the user for confirmation. Fetch logs, interpret, fix immediately. No user-facing handoffs (rollout triggering belongs to oncall-iteration-loop). If you can't map a pattern with confidence, escalate to horizon-agentic-reviewer.

Trigger

Oracle ≠ 1.0/passed; No-Op score > 0; grader exception; feedback: null (setup crash); or before editing setup/solution/grader after a validation failure. NOT for rollout failures — use oncall-score-tuner.

Task Identity

Field	Value
UUID	<TASK_ID>
Slug / path	oncall-schedule-coverage-collapse / tasks/oncall-schedule-coverage-collapse/
Subscores	coverage_restored (0.5) + monitoring_governance (0.5)
Venv	source /Users/mac/Documents/tasks/horizon_env/bin/activate

Root-only files the grader reads: /opt/grader-scenario.json (keys reconciler, env_reconciler, canary, canary_runtime), /opt/<grader-artefact>.txt (mirror of agent-readable /etc/oncall/<artefact>.txt). If /opt/<grader-artefact>.txt is missing the grader SKIPS the <metadata-key> gate (no deadweight) — a "governance always-1 with the anchor silently skipped" means the setup write failed; check output.txt.

Grader feedback = coverage_restored: <detail>; monitoring_governance: <detail> — read both halves.

Result shapes

score	passed	feedback	meaning
0	false	null	setup/container crash before grader → output.txt
0	false	non-null	both failed
0.5	false	non-null	one passed, one failed
1.0	true	non-null	both passed

No-Op: anything but score: 0 is a problem.

Step 1 — Fetch + read

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate-logs -a oracle tasks/oncall-schedule-coverage-collapse 2>&1
cat tasks/oncall-schedule-coverage-collapse/.validation/*/result.json 2>/dev/null | python3 -m json.tool

Step 2 — Map symptom → cause → fix

Branch A. feedback: null / score: 0 — setup crash

Pattern in output.txt	Cause	Fix
hang at Waiting for bleater namespace	base image didn't create bleater	platform issue; not ours
namespaces "bleater-env" ... Forbidden / create error	setup can't create bleater-env	setup runs as root and does `kubectl get ns bleater-env
k3s is not ready after 180 seconds	ENABLE_ISTIO_BLEATER=true	remove it
Forbidden on kubectl get nodes	rbac yaml not copied	restore COPY data/ubuntu-user-rbac.yaml
failed to pull image for a drift/canary pod	external registry	use ${SIDECAR_IMAGE}
setup.sh: line N: syntax error	bash error	bash -n setup.sh

Branch B. No-Op score > 0 — false positive

Passed on No-Op	Why wrong	Fix
coverage_restored=1	the broken state didn't apply, OR drift didn't deploy so the schedule/escalation were never (re)broken	confirm oncall-schedule-primary has carol/dave/UNASSIGNED + oncall-escalation-secondary → incidents-archived + all 3 drift deploys exist
monitoring_governance=1	drift didn't deploy, OR the schedule already has coverage-tier/<metadata-key> at setup time, OR the install-id mirror wasn't written	confirm setup leaves the schedule WITHOUT the label/anchor, writes /opt/<grader-artefact>.txt, and deploys all drift

Branch C. Oracle 0.5 — coverage_restored=0 (governance passed)

Feedback fragment	Cause	Fix in solution.sh
primary rotation has an unassigned slot / owned by <user> (inactive...)	the kube-system reconciler or canary re-applied the broken schedule	stop BOTH (annotation discovery for the reconciler; slot-pinner container for the canary) BEFORE re-applying the corrected schedule with active owners
override ... still references deactivated user	overrides not cleaned	set oncall-overrides overrides.json to [] (or reassign to active) — overrides are NOT drifted, so this sticks once applied
secondary escalation points at channel ... not active	env reconciler re-applied the archived channel	stop the bleater-env reconciler (annotation) BEFORE re-applying policy.json with an active channel (incidents)

Branch D. Oracle 0.5 — monitoring_governance=0 (coverage passed)

Feedback fragment	Cause	Fix in solution.sh
schedule missing required label coverage-tier=enforced	corrected schedule lacks the marker	add label bleater.io/coverage-tier: enforced
schedule does not anchor to the platform install id	no annotation == install id	read /etc/oncall/<artefact>.txt, set an annotation (any key) to that value
no schedule-gap / unresolvable-slot monitoring rule found	rule CM missing or not a real rule	create a CM with - alert:/expr: referencing the on-call schedule + a gap/coverage/unresolvable signal
active drift ... <name>@kube-system / @bleater-env	a reconciler still running	discover by annotation, delete/scale to 0 (incl. bleater-env)
<canary> mutation capability not fully neutralized	slot-pinner sidecar AND runtime RBAC present	delete the canary OR remove <canary>-runtime RBAC

Branch E. Oracle 0 non-null — both failed

Read both halves; follow C and D in parallel. Persists after fixes → re-run No-Op to isolate setup contamination.

Branch F. Grader exception

All kubectl/JSON access is guarded (kubectl_json, cm_data_json, .get). If one crept past, wrap it; treat failure as subscore=0. Not a hard-constraint item.

Step 3 — Pre-fix safety

1. Dockerfile + bleater-env invariant hold (guardian Step 0).
2. bash -n setup.sh && bash -n solution.sh && py_compile grader.py.
3. Subscore-difficulty change → diagnose with oncall-score-tuner first.
4. Articulate the why.

Fix type	Hand off
setup fault-injection / solution bug / grader exception	oncall-task-guardian Step 3 (push)
subscore difficulty	oncall-score-tuner first
unmappable	horizon-agentic-reviewer

Known non-issues (do NOT fix)

Pattern	Why
v1_feedback + behavior_in_task_documentation FAIL	reviewer can't read the baseline-config heredoc → calls the coverage-tier / <metadata-key> gates "undocumented". They ARE the variance mechanism — accepted
governance fails because agents only audited bleater	the kube-system + bleater-env discovery axis working as intended

Verification cheat sheet

Hypothesis	Probe
"solution stops drift before re-applying state"	read solution.sh — the annotation/slot-pinner discovery+delete precede the schedule/escalation re-apply
"solution sets label + anchor + gap-rule + active channel + empty overrides"	`grep -E "coverage-tier
"grader gap-rule gate intact"	`grep -E "is_rule
"wait_consistent unchanged"	grep wait_consistent grader.py — coverage passes=5,gap=8,timeout=220; governance passes=5,gap=8,timeout=200