Oncall score tuner

oncall-score-tuner

QC Spec compliance — assert before every tuning edit

Any difficulty/variance lever you pull must STILL satisfy the QC Spec (.claude/QC-SPEC.md). Before recommending a change, assert it does not breach:

• No Binary Bundling — never recover variance by AND-ing more heterogeneous checks into one subscore.
• Incremental Rewards / No Dead Weights — no subscore left always-0 or always-1; every lever keeps both subscores varying.
• Functional Subscore Variance — at least one functional subscore takes ≥2 distinct values across a batch.
• Equal Weights — never rebalance weights to hit a number.
• Functional Tests as Subscores / No Answer Leakage — don't convert a functional subscore into static matching, and don't leak the answer to manufacture passes.

A lever that lowers avg by breaching any of these is NOT a valid fix — pick a different lever.

Execution Mode — FULLY AUTONOMOUS

NEVER pause for confirmation. Pick the best path and execute it.

Goal

Keep avg < <TARGET_MEAN>, both subscores varying, decorrelated, no deadweight. Smallest reversible edits. Invoked on: avg ≥ 0.50; a subscore deadweight; correlation collapse (no 0.5 rollouts); oncall-task-guardian Step 6 hand-off. NOT for Oracle failures → oncall-validation-debugger.

Task Identity

Field	Value
UUID	<TASK_ID>
Scoring	coverage_restored × 0.5 + monitoring_governance × 0.5
Target	avg < <TARGET_MEAN>, both in {0,1}`, NOT perfectly correlated

Status: not yet rollout-tuned. Watch the first batch for correlation collapse (the predicted failure mode).

How difficulty is constructed

OnCall state simulated as ConfigMaps in bleater. The agent repairs coverage + escalation + overrides, anchors metadata, adds a gap-detection rule, and stops three cross-namespace drift sources.

The shared surface

Drift	Namespace	Re-applies	Cadence	Discover by
reconciler	kube-system	broken schedule (dead/UNASSIGNED slots, strips metadata)	5s	bleater.io/purpose annotation
env reconciler	bleater-env	broken escalation (archived channel)	20s	annotation
canary slot-pinner	bleater	broken schedule again	11s	container name

Both subscores depend on stopping this set → correlation risk. The kube-system reconciler/canary re-break the schedule (coverage layer 1 + governance metadata layers) and are governance-graded; the env reconciler re-breaks escalation (coverage layer 3) and is governance-graded.

coverage_restored axis (wait_consistent 5/8/220)

slots all active owners (no gap), no inactive override, escalation channel active. Independent decorrelator: the overrides cleanup — overrides are NOT drifted, so an agent who stops all drift still fails coverage if they leave carol/dave overrides.

monitoring_governance axis (wait_consistent 5/8/200)

coverage-tier label; <metadata-key> annotation (== /etc/oncall/<artefact>.txt, grader reads /opt/<grader-artefact>.txt, skipped if missing); gap/unresolvable rule (real - alert:/expr: + on-call subject + gap signal); kube-system + bleater-env reconcilers replicas=0; canary neutralized. Independent decorrelators: the gap-rule construction (agents who hand-wave a CM with "gap" in a comment fail the - alert:/expr: requirement) and the <metadata-key> thoroughness gate (read /etc/oncall/, ~40%-catch on the sibling pattern).

Structural deadweight check (do FIRST if governance always-0)

Confirm bleater-env reachability — ALLOWED_NAMESPACES must include bleater-env (it ships set; don't drop it) or the env reconciler is unstoppable → governance deadweight-0. Also confirm /opt/<grader-artefact>.txt was written (else the anchor gate silently skips, making governance easier, not harder).

Four rules

1. Preserve variance. Stuck-at-one = broken. Always-0 → bleater-env reachability / install-id write. Always-1 → too discoverable.
2. Break correlation. Prefer a lever moving ONE subscore: coverage → overrides; governance → gap-rule / <metadata-key>. A drift-discoverability lever moves both — use only to shift avg.
3. Functional > static. The gap-rule (real PromQL keyed by schedule) and <metadata-key> (a value the agent must fetch from /etc/oncall/) are durable. Never require an exact CM name; never leak the exact label/value to task.yaml.
4. Minimal safe edits.

Tuning order

Level 1 — setup.sh

Shared drift (move BOTH — shift avg only): reconciler 5s / env 20s / canary 11s cadences (faster = harder, floors 5/10/8); pool-name innocuousness (keep annotation-discoverable). Decorrelating (preferred): strengthen/weaken the overrides fault (coverage-only); the install-id artefact + baseline pointer wording (governance-only thoroughness); the gap-rule expectation phrasing (governance-only).

Level 2 — grader.py

wait_consistent (ceilings gap 8s; coverage timeout floor 180, governance 150). The gap-rule strictness (subject + rule-key + gap-signal) is the governance construction gate — keep functional, name-agnostic. Don't add a check that just matches a baseline-copyable value.

Level 3 — task.yaml (last resort)

Allowed: strengthen/weaken the "more than one namespace" hint; the "read baseline in full / inspect /etc/oncall" pointer. Forbidden: naming a drift controller, the annotation value, the exact metadata key/value, or itemizing the fix.

Diagnosis guide

Symptom	Cause	Lever
no 0.5 rollouts (correlation)	drift is the only hard thing	lean on the overrides (coverage-only) + gap-rule/<metadata-key> (governance-only) decorrelators; don't pull a shared-drift lever
coverage_restored always 1	drift easy + agents fix overrides	speed env/canary cadence; sharpen the overrides fault
coverage_restored always 0 (Oracle 1.0)	a drift source out-races the window	confirm cadences ≥ floor; verify bleater-env reachable
monitoring_governance always 1	gap-rule + anchor too easy	keep the gap-rule strict; keep coverage-tier/anchor buried in baseline
monitoring_governance always 0 (Oracle 1.0)	bleater-env unreachable OR install-id mirror missing	structural — fix Dockerfile/setup, not grader
avg 0.60–0.70	slightly easy	tighten a decorrelating gate, not a shared one

Roles

• R1 Rollout Analyst — run the iteration-loop script, read ≥2 transcripts, classify (deadweight / healthy / correlated).
• R2 Setup Drift Specialist — which drift do failing agents miss? (bleater-env env-reconciler is the sneaky cross-namespace one.)
• R3 Coverage Specialist — overrides + escalation; the overrides cleanup is the coverage-only decorrelator.
• R4 Governance Specialist — gap-rule + <metadata-key> + coverage-tier; rule out bleater-env reachability + install-id write first.
• R5 Coupling Reviewer (critical) — does the edit move ONLY shared drift? If yes → REJECT for decorrelation work; propose an overrides/gap-rule/anchor alternative.
• R6 QC Reviewer — expect 18 substantive pass + v1_feedback/behavior_in_task_documentation noise FAIL. Reject naming keys/values/drift in task.yaml; reject an exact-CM-name rule check.
• R7 Final Recommender — 8-point block (root cause / why avg shifted / smallest fix / rollout+correlation impact / QC impact / risk / files / lines). Then → oncall-iteration-loop → oncall-task-guardian.

Operating procedure

Confirm trigger → read grader/setup/task + rollouts + ≥2 transcripts → diagnose (R1) → pick lever (prefer decorrelating) → refine (R3/R4) → review (R5 + R6) → recommend (R7) → NEVER apply here; hand to the loop/guardian.

Hard constraints

No ENABLE_ISTIO_BLEATER; keep ALLOWED_NAMESPACES="kube-system,bleater-env"; keep the rbac COPY; never name drift/keys/values in task.yaml/baseline; keep the gap-rule functional & name-agnostic; air-gapped images; wait_consistent ceilings 5/8s (coverage 220 / governance 200); don't simplify solution.sh.

Preferred outcomes

Healthy: 0.0/0.5/1.0 present; avg <FLOOR>–0.55; both vary AND decorrelated. Unhealthy: deadweight; avg > <TARGET_MEAN>; perfect correlation; structural impossibility (bleater-env unreachable, install-id mirror missing). Two cycles chasing the same Unhealthy pattern → halt and summarize.