Oncall task guardian

oncall-task-guardian

Step 0 — QC Spec design gate (READ FIRST, every iteration)

Before editing ANY task file and again before every push, read the QC Spec — .claude/QC-SPEC.md. It is the priority-ordered rubric (Critical → Error → Legitimacy) horizon check-quality and reviewers grade against.

Confirm the intended/current design does not violate any Critical or Error item. The ones this task repeatedly trips:

• No Binary Bundling — never AND heterogeneous checks (metadata + drift Deployments + schedule shape) into one binary subscore. Each subscore = ONE cohesive functional outcome.
• Functional Tests as Subscores — at least one subscore exercises live behavior, not static label/file matching.
• Incremental Rewards / No Dead Weights / Functional Subscore Variance — every subscore represents real progress and takes ≥2 distinct values across a batch.
• Equal Weights — subscores equally weighted.
• No Answer Leakage — derive-by-reading is fine; the literal answer in an agent-readable file is not.

If a change would breach a Critical/Error item, stop and rework it before pushing.

Execution Mode — FULLY AUTONOMOUS

NEVER ask the user for confirmation on any terminal command, edit, push, validation, or live-probe. The ONLY user-facing handoff is Step 4c: telling the user to trigger the rollout batch in the Horizon UI (you cannot trigger rollouts from the CLI). Everything else runs without asking. If a decision has two valid paths, pick the safer one.

Goal

Keep oncall-schedule-coverage-collapse shippable every push: setup.sh runs cleanly, Oracle scores 1.0, rollouts average < <TARGET_MEAN> with both subscores varying AND decorrelated. The only skill that pushes the task; the front door for the other oncall-* skills.

Task Identity

Field	Value
Task UUID	<TASK_ID>
Task slug / id	oncall-schedule-coverage-collapse
Local task path	tasks/oncall-schedule-coverage-collapse/
Horizon root	/Users/mac/Documents/tasks
Venv	source /Users/mac/Documents/tasks/horizon_env/bin/activate
Scoring	coverage_restored × 0.5 + monitoring_governance × 0.5 — both binary
Target avg	rollout avg < <TARGET_MEAN>, both subscores varying [0,1]`, decorrelated (0.5 rollouts exist)
Mini-batch	<TASK_ID>

Status: built, locally Oracle-traced (k8s-config simulation, graded by shape). NOT yet hosted-Oracle / rollout-tuned — variance numbers are targets.

How scoring actually works (read once, internalise)

Always derive constants from grader.py. Correct as of grader.py v1.

The OnCall world is simulated as ConfigMaps in bleater (no real Grafana OnCall on this image): oncall-users (active/inactive), oncall-schedule-primary (slots.json: {window,owner}), oncall-overrides (overrides.json), oncall-escalation-secondary (policy.json: {channel}), mattermost-channels (channels.json: active/archived). The grader scores config shape. Two binary subscores, each 0.5:

• coverage_restored = 1 (check_coverage_restored, wait_consistent(passes=5, gap=8s, timeout=220s)) when ALL hold:
  • every oncall-schedule-primary slot owner is active in oncall-users (no UNASSIGNED/gap, no inactive/unknown owner)
  • no oncall-overrides entry references an inactive user
  • oncall-escalation-secondary's channel is active in mattermost-channels (not archived/unknown)
• monitoring_governance = 1 (check_monitoring_governance, wait_consistent(passes=5, gap=8s, timeout=200s)) when ALL hold:
  • the schedule ConfigMap carries label bleater.io/coverage-tier=enforced (baseline-documented only — thoroughness gate)
  • the schedule has an annotation whose VALUE equals /etc/oncall/<artefact>.txt (install anchoring; grader reads root-only /opt/<grader-artefact>.txt; skipped if that file is missing → no deadweight). Key is the agent's choice.
  • a gap/unresolvable monitoring rule exists — ANY ConfigMap in bleater whose data has an on-call subject (oncall/schedule/rotation), a real rule key (- alert:/- record:/expr:), AND a gap signal (gap/coverage/unresolvable/no owner/uncovered)
  • the kube-system reconciler (replicas=0) AND the bleater-env reconciler (replicas=0)
  • the canary slot-pinner mutation path neutralized — sidecar removed OR <canary>-runtime SA/Role/RoleBinding removed

score=0 neither; 0.5 one; 1.0 both.

Drift controllers (re-apply the broken state)

Names randomized (pool stem + numeric suffix); grader reads them from /opt/grader-scenario.json (root-only, keys reconciler, env_reconciler, canary, canary_runtime). solution.sh discovers by the bleater.io/purpose: "oncall coverage reconciliation" annotation + the slot-pinner container name.

Drift source	Namespace	Re-applies	Cadence	Discover by
reconciler	kube-system	broken schedule (carol/dave/UNASSIGNED slots, strips metadata)	5s	annotation
env reconciler	bleater-env	broken escalation (archived channel)	20s	annotation
canary slot-pinner	bleater	broken schedule again	11s	slot-pinner container; RBAC <canary>-runtime

Coupling note (read before tuning)

Shared surface: the kube-system reconciler + canary re-break the schedule (fails coverage_restored layer 1 AND, by stripping metadata, monitoring_governance layers 0–1) and are governance-graded at replicas=0. The env reconciler re-breaks escalation (coverage layer 3) and is governance-graded. So drift couples the two subscores. Decorrelators: coverage_restored-only = the overrides cleanup (overrides are NOT drifted); monitoring_governance-only = the gap-rule construction + the <metadata-key> thoroughness gate.

---

Step 0 — Dockerfile + namespace invariants (ALWAYS before push)

cat tasks/oncall-schedule-coverage-collapse/Dockerfile

Invariant	Required	Why
ENABLE_ISTIO_BLEATER	must NOT appear	k3s crash → score: 0, feedback: null
ENV ALLOWED_NAMESPACES="kube-system,bleater-env"	both present	the bleater-env reconciler (governance layer 3) is unstoppable without bleater-env access → structural always-0. This task ships BOTH already; do not drop bleater-env.
COPY data/ubuntu-user-rbac.yaml ...	present	DO-NOT-CHANGE block needs the node ClusterRole; setup aborts otherwise

Verify on the live container (Step 4b) that kubectl get deploy -n bleater-env is NOT Forbidden for the ubuntu agent.

Step 1 — Local syntax + anatomy

cd /Users/mac/Documents/tasks
bash -n tasks/oncall-schedule-coverage-collapse/setup.sh    && echo "setup.sh OK"
bash -n tasks/oncall-schedule-coverage-collapse/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/oncall-schedule-coverage-collapse/grader.py && echo "grader.py OK"
source horizon_env/bin/activate
horizon check-anatomy tasks/oncall-schedule-coverage-collapse 2>&1

Step 2 — Local quality

horizon check-quality tasks/oncall-schedule-coverage-collapse 2>&1 | tail -30

Before push, walk the QC-SPEC.md Critical + Error list against the current grader.py + task.yaml and record pass/fail per item. Every item must pass OR be on the accepted-noise list with a one-line reason. A new failing item that is NOT accepted-noise blocks the push. Expected ~18/20 substantive pass; verdict may be FAIL driven by v1_feedback + behavior_in_task_documentation — both are the documented heredoc-invisibility noise (the reviewer can't read the baseline-config heredoc, so it calls the coverage-tier / <metadata-key> gates "undocumented"). These gates ARE the rollout-variance mechanism — do NOT fix by naming the keys/values in task.yaml (collapses monitoring_governance to always-1). Any OTHER check failing is a real signal.

Step 3 — Push

horizon tasks push tasks/oncall-schedule-coverage-collapse 2>&1   # record Version: NNN

Step 4 — Validation handoff

4a. Oracle (~10–15 min): pass = 1.0 + both subscores 1. Else → oncall-validation-debugger. 4b. Live red-team via horizon-agentic-reviewer on UUID <TASK_ID>. Verify (as ubuntu):

Hypothesis	Probe
bleater-env reachable	kubectl get deploy -n bleater-env — not Forbidden
broken state live	kubectl get cm oncall-schedule-primary -n bleater -o jsonpath='{.data.slots\.json}' shows carol/dave/UNASSIGNED; oncall-escalation-secondary → incidents-archived
all 3 drift sources exist + annotated	`kubectl get deploy -A -o json
install artefact present + agent-readable	cat /etc/oncall/<artefact>.txt works as ubuntu; /opt/<grader-artefact>.txt NOT readable as ubuntu
drift names + the exact metadata keys NOT leaked in baseline	`grep -iE "reconciler
no-op agent fails both subscores	reset, run grader — both 0

4c. Tell the user, verbatim:

"Oracle passed at version N and live red-team is clean. Please trigger a rollout batch for version N in the Horizon UI."

Step 5 — Monitor rollouts

horizon rollouts pull --version NNN tasks/oncall-schedule-coverage-collapse 2>&1

Then the oncall-iteration-loop Phase 3 script.

Metric	Target
avg	< 0.50 (and not < <FLOOR> — over-hard)
coverage_restored	both 0 and 1; pass ~25–60%
monitoring_governance	both 0 and 1; pass ~20–50%
decorrelation	some 0.5 rollouts

Step 6 — Tuning decision

Observation	Hand-off
Oracle < 1.0	oncall-validation-debugger
avg ≥ 0.50 OR a subscore deadweight	oncall-score-tuner
no 0.5 rollouts (correlation collapse)	oncall-score-tuner (lean on overrides-only + gap-rule/<metadata-key> decorrelators)
monitoring_governance always 0	suspect bleater-env reachability OR the install-id artefact write — verify live first

---

Hard constraints

Constraint	Why
No ENABLE_ISTIO_BLEATER=true	k3s crash
Keep ALLOWED_NAMESPACES="kube-system,bleater-env" (both)	bleater-env reconciler unstoppable otherwise → governance always-0
Keep COPY data/ubuntu-user-rbac.yaml path	setup aborts otherwise
Never name drift controllers / the bleater.io/purpose value / the exact metadata keys+values in task.yaml or baseline	collapses monitoring_governance to always-1
Keep the gap-rule check name-agnostic (subject + rule-key + gap-signal), never require an exact CM name	static dead-weight otherwise
Air-gapped images only (${SIDECAR_IMAGE})	failed pull → ghost drift
wait_consistent ceilings passes=5, gap=8s (coverage 220 / governance 200)	above → flake to dead-zero
Don't simplify solution.sh to make Oracle pass	it's the oracle standard, not a tuning lever

Skill hand-off map

oncall-task-guardian (you are here)
├── Step 4a fails → oncall-validation-debugger
├── Step 4b       → horizon-agentic-reviewer
├── Step 5 fails  → oncall-score-tuner
└── Step 6 loops  → oncall-iteration-loop (orchestrator)