Fanout cve iteration loop

fanout-cve-iteration-loop

Execution Mode — FULLY AUTONOMOUS

NEVER pause for confirmation. Run every phase, command, and fix without asking. The only manual step is the user triggering the eval batch in the Horizon UI.

Goal

Drive fanout-cve-rollout to shippable. This is a reward-shaped task (5 independent subscores × 0.2) — the exit bar is Oracle 1.0 + a dense, correct partial-credit gradient, NOT an avg < <TARGET_MEAN> variance target.

Trigger

"Iterate the fanout-cve task until it passes"; "push and validate fanout-cve-rollout"; anything implying repeated push → validate → analyze → fix. Single steps → invoke the specialist directly.

Task Identity

Field	Value
UUID	<TASK_ID>
Slug / path	fanout-cve-rollout / tasks/fanout-cve-rollout/
Horizon root	/Users/mac/Documents/tasks
Venv	source /Users/mac/Documents/tasks/horizon_env/bin/activate

Who does what

The agent executes everything except triggering the eval batch in the Horizon UI.

Exit conditions (all, same version)

1. Oracle passed: true, score: 1.0, all five subscores =1
2. Live red-team: no BLOCKING findings — especially the stale-env drift live (CM peer, pod env full), grader truth root-only, a no-op agent scoring 0, and a ConfigMap-only fix scoring exactly 0.2 with the roll-the-pods hint
3. Rollout gradient: the score histogram spans {0,0.2,…,1.0}; no subscore deadweight; a fair share reach 1.0
4. pods_rolled_live_env is reachable (some agents pass it) and instructive (a meaningful share miss it)
5. Local quality: 18 substantive pass (v1_feedback may FAIL as accepted noise)

Short of all → continue.

Loop limit

Max 5 push cycles without human review, then summarize.

No-Op default

Oracle only. Run No-Op manually only on feedback: null/score: 0 (setup-crash isolation) OR once to confirm a no-op agent scores 0 across all five subscores.

The loop

PHASE 0  Pre-push (fanout-cve-task-guardian Step 0–2): Dockerfile + bash -n + py_compile + anatomy + quality
PHASE 1  Push → version NNN
PHASE 2  Oracle → 1.0 + all five=1 → PHASE 2.5 ; else → fanout-cve-validation-debugger → fix → PHASE 0
PHASE 2.5 Live red-team (horizon-agentic-reviewer) → clean → ask user to trigger batch ; BLOCKING → fix → PHASE 0
PHASE 3  Rollout gradient analysis → dense+correct gradient → DONE ; else → fanout-cve-score-tuner → PHASE 0

Phase 0 — Pre-push

cd /Users/mac/Documents/tasks
echo "=== Dockerfile ===" && cat tasks/fanout-cve-rollout/Dockerfile   # no ENABLE_ISTIO_BLEATER; COPY data/ubuntu-user-rbac.yaml
bash -n tasks/fanout-cve-rollout/setup.sh    && echo "setup.sh OK"
bash -n tasks/fanout-cve-rollout/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/fanout-cve-rollout/grader.py && echo "grader.py OK"
source horizon_env/bin/activate
horizon check-anatomy tasks/fanout-cve-rollout 2>&1
horizon check-quality tasks/fanout-cve-rollout 2>&1 | tail -25

Phase 1 — Push

horizon tasks push tasks/fanout-cve-rollout 2>&1   # record Version: NNN

Phase 2 — Oracle (poll, don't trust --wait)

horizon tasks validate -m hosted -a oracle tasks/fanout-cve-rollout 2>&1
BUILD_ID="val-31ba76bf-<TIMESTAMP>"
RESULT_PATH="tasks/fanout-cve-rollout/.validation/${BUILD_ID}/result.json"
for attempt in $(seq 1 30); do
  sleep 60
  horizon tasks validate-logs -a oracle tasks/fanout-cve-rollout >/dev/null 2>&1
  STATUS=$(python3 -c "import json; print(json.load(open('$RESULT_PATH')).get('status','unknown'))" 2>/dev/null)
  if [ "$STATUS" != "running" ] && [ -n "$STATUS" ]; then
    echo "Oracle done after ${attempt} min, status=$STATUS"; cat "$RESULT_PATH" | python3 -m json.tool; break
  fi
  echo "[poll ${attempt}/30] status=$STATUS"
done

Pass: 1.0 + all five subscores 1. Run as a background bash task. Else → fanout-cve-validation-debugger. Common oracle bug: rolling pods before reverting the ConfigMap → new pods snapshot peer → pods_rolled_live_env=0. Revert CM first, then roll.

Phase 2.5 — Live red-team

Invoke horizon-agentic-reviewer on UUID <TASK_ID>. Confirm (probes in fanout-cve-task-guardian Step 4b): stale-env drift live (CM peer vs pod env full); deployment on vulnerable 1.4.2; /opt/grader-cve.json not agent-readable; helm-values/argocd start broken; a no-op agent scores 0; a ConfigMap-only fix scores 0.2 + hint. Auto-invoked. BLOCKING → fix → re-loop. Skip only for a pure copy-edit. VM unreachable → note risk, proceed.

Tell the user (verbatim):

"Oracle passed at version N and live red-team is [clean | skipped: reason]. Please trigger a rollout batch for version N in the Horizon UI."

Phase 3 — Rollout gradient analysis

horizon rollouts pull --version NNN tasks/fanout-cve-rollout 2>&1

Poll until ≥5; read 2 transcripts (one partial, one full) before the script.

import json, glob
from collections import Counter, defaultdict
VERSION="NNN"; TASK="fanout-cve-rollout"
files=glob.glob(f"tasks/{TASK}/.rollouts/v{VERSION}/*.json")
scores=[]; sub=defaultdict(list)
for f in files:
    d=json.load(open(f)); scores.append(round(d["score"],2))
    try:
        for k,v in json.loads(d.get("grade_result","{}")).get("subscores",{}).items(): sub[k].append(v)
    except Exception: pass
if not scores: print("No rollouts — trigger a batch first")
else:
    n=len(scores)
    print(f"N={n} avg={sum(scores)/n:.2f} full_fix_rate={sum(1 for s in scores if s>=0.99)/n:.0%}")
    print("  score histogram:", dict(sorted(Counter(scores).items())))
    for k,vs in sub.items():
        pr=sum(vs)/len(vs); tag="DEAD" if len(set(vs))==1 else "ok"
        print(f"  {k}: pass={pr:.0%} {tag}")
    dead=[k for k,vs in sub.items() if len(set(vs))==1]
    if dead: print("DEADWEIGHT:", dead, "— fanout-cve-score-tuner")
    elif len(set(scores))<=2: print("NO PARTIAL CREDIT (only 2 score values) — fanout-cve-score-tuner")
    else: print("DONE: dense gradient, no deadweight")

Result	Action
dense gradient {0..1.0}, no deadweight, some reach 1.0	DONE
a subscore deadweight (always 0 or 1)	fanout-cve-score-tuner
only 0.0/1.0 appear (no partial credit)	fanout-cve-score-tuner (subscores coupled)
nobody reaches 1.0	fanout-cve-validation-debugger (likely the revert-then-roll ordering) then re-Oracle
<5 rollouts	poll again

Decision rules (Phase 2)

Oracle	Action
1.0, all five=1	Phase 2.5
feedback: null/0	fanout-cve-validation-debugger Branch A
partial < 1.0	Branch C — read which subscore is 0
exception	Branch D

When to stop for human review

5 cycles without progress; a tuner fix would violate a hard constraint (anti-hack, live-env exec, independence, medium difficulty); nobody reaches 1.0 despite a correct-looking solution (reviewer live probe).

Skill dependency map

fanout-cve-iteration-loop (orchestrator)
├── fanout-cve-task-guardian        Phase 0 pre-push, Phase 1 push, Phase 5 monitoring
├── fanout-cve-validation-debugger  Phase 2 failure interpretation
├── fanout-cve-score-tuner          Phase 3 reward-gradient diagnosis
└── horizon-agentic-reviewer        Phase 2.5 live red-team