Maddy iteration loop

maddy-iteration-loop

Execution Mode — FULLY AUTONOMOUS

NEVER call ask_user or pause for confirmation at any point. Run every phase, every bash command, and every fix without asking permission. If a decision has two valid paths, pick the safer one and proceed. The user triggered this skill precisely to avoid being prompted — do not interrupt them.

Goal

Drive the maddy-delivery-split-brain task from "needs work" to "shippable" without human intervention except for triggering the rollout batch in the Horizon UI. The loop is done when all exit conditions hold; until then, every cycle picks the right specialist skill (validation-debugger, score-tuner, horizon-agentic-reviewer) for the symptom and applies the smallest safe fix.

This is the front-door skill for "iterate this task to a passing state." The other three maddy-* skills are pure capabilities; this skill orchestrates them.

Trigger

Use this skill when asked to:

• "Iterate the maddy task until it passes"
• "Push and tune until avg < <TARGET_MEAN>"
• "Run the full loop on maddy-delivery-split-brain"
• Anything implying repeated push → validate → analyze → fix cycles

For single-step requests (just push, just analyze rollouts, just debug a validation), invoke the relevant specialist skill directly.

Task Identity

Field	Value
Task UUID	<TASK_ID>
Task slug	maddy-delivery-split-brain
Local path	tasks/maddy-delivery-split-brain/
Horizon root	/Users/mac/Documents/tasks
Venv	source /Users/mac/Documents/tasks/horizon_env/bin/activate

---

Who does what

The agent (you) executes every step except 2.5c. Never tell the user to run push, validate, or any CLI command.

Step	Who
Edit files (setup.sh, grader.py, solution.sh, task.yaml, Dockerfile)	Agent
Pre-push syntax + anatomy + quality checks	Agent
Push (horizon tasks push)	Agent
Oracle validation	Agent
Live red-team	Agent (invokes horizon-agentic-reviewer)
Pull + analyze rollouts	Agent
Diagnose tuning / validation fixes	Agent (invokes maddy-score-tuner / maddy-validation-debugger)
Trigger the eval batch in the Horizon UI	User (the only manual step)

---

Exit conditions

The loop is DONE when all hold on the same version:

1. Oracle: passed: true, score: 1.0, both subscores =1
2. Live red-team: no BLOCKING findings — especially bleater-env reachability (see maddy-task-guardian Step 4b)
3. Rollout avg: < <TARGET_MEAN>
4. delivery_consistency: varies (both 0 and 1 appear)
5. observability_governance: varies (both 0 and 1 appear)
6. The two subscores are not perfectly correlated (at least some 0.5 rollouts appear, not only 0.0 and 1.0)
7. Local quality: 18 substantive checks pass (the v1_feedback section may FAIL as accepted noise)

Anything short of all seven → loop continues.

Loop limit

Do not exceed 5 push cycles without human review. After 5 cycles with no measurable progress on a specific failure mode, stop and summarize what was tried and what's still failing.

No-Op default

Do NOT run No-Op as part of the loop. Oracle only. Oracle exercises the same setup.sh path. Run No-Op manually only when Oracle returns feedback: null / score: 0 with no grader detail (setup-crash isolation).

---

The loop

START
  ▼
PHASE 0: Pre-push checks ─────────────────────────────────────────
  • Dockerfile + namespace invariants (maddy-task-guardian Step 0)
    — incl. bleater-env reachability risk
  • bash -n setup.sh && bash -n solution.sh && py_compile grader
  • horizon check-anatomy (must pass clean)
  • horizon check-quality (18 substantive pass; v1_feedback may FAIL)
  If anything else fails → fix → re-run PHASE 0
  ▼
PHASE 1: Push ────────────────────────────────────────────────────
  horizon tasks push → record version NNN
  ▼
PHASE 2: Oracle validation ───────────────────────────────────────
  → 1.0 + both subscores=1 → PHASE 2.5
  → anything else → maddy-validation-debugger → fix → PHASE 0
  ▼
PHASE 2.5: Live red-team ──────────────────────────────────────────
  invoke horizon-agentic-reviewer on the UUID
  → no BLOCKING finding → ask user to trigger rollout batch
  → BLOCKING (esp. bleater-env unreachable) → fix → PHASE 0
  ▼
PHASE 3: Rollout analysis ─────────────────────────────────────────
  pull ≥5 rollouts; read 2 transcripts (1 pass,1 fail); run script
  → avg < <TARGET_MEAN> AND both vary AND not perfectly correlated → DONE
  → else → maddy-score-tuner → fix → PHASE 0
  ▼
DONE

---

Phase 0 — Pre-push commands

cd /Users/mac/Documents/tasks

echo "=== Dockerfile ===" && cat tasks/maddy-delivery-split-brain/Dockerfile
#   Must NOT contain: ENABLE_ISTIO_BLEATER
#   Must contain:     ALLOWED_NAMESPACES (incl. kube-system; verify bleater-env reachability on live container)
#   Must contain:     COPY data/ubuntu-user-rbac.yaml

bash -n tasks/maddy-delivery-split-brain/setup.sh    && echo "setup.sh OK"
bash -n tasks/maddy-delivery-split-brain/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/maddy-delivery-split-brain/grader.py && echo "grader.py OK"

source horizon_env/bin/activate
horizon check-anatomy tasks/maddy-delivery-split-brain 2>&1
horizon check-quality tasks/maddy-delivery-split-brain 2>&1 | tail -30

Phase 1 — Push

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/maddy-delivery-split-brain 2>&1
# Record: "✓ New version pushed successfully! Version: NNN"

Phase 2 — Oracle validation

Do NOT rely on --wait alone (TTY spinners buffer and the foreground wait can hang after the hosted run finished). Poll via validate-logs:

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

horizon tasks validate -m hosted -a oracle tasks/maddy-delivery-split-brain 2>&1
# Capture Build ID: "val-1c67bc57-<timestamp>" (or derive from the .validation/ subdir name)
BUILD_ID="val-1c67bc57-<TIMESTAMP_FROM_OUTPUT>"
RESULT_PATH="tasks/maddy-delivery-split-brain/.validation/${BUILD_ID}/result.json"

for attempt in $(seq 1 30); do
  sleep 60
  horizon tasks validate-logs -a oracle tasks/maddy-delivery-split-brain >/dev/null 2>&1
  STATUS=$(python3 -c "import json; print(json.load(open('$RESULT_PATH')).get('status','unknown'))" 2>/dev/null)
  if [ "$STATUS" != "running" ] && [ -n "$STATUS" ]; then
    echo "Oracle completed after ${attempt} min, status=$STATUS"
    cat "$RESULT_PATH" | python3 -m json.tool
    break
  fi
  echo "[poll ${attempt}/30] status=$STATUS"
done

Pass: score: 1.0, passed: true, both delivery_consistency=1 and observability_governance=1. Run via a background bash task (run_in_background: true) and let the harness notify — don't foreground-wait in a 2-minute Bash call.

Anything else → invoke maddy-validation-debugger (do NOT edit files directly first).

Phase 2.5 — Live red-team

Invoke horizon-agentic-reviewer against UUID <TASK_ID>. The reviewer spins up a live container, runs horizon setup, executes setup.sh manually (horizon setup does NOT — see horizon-agentic-reviewer Step 3.5), probes as ubuntu, runs the 24-point checklist.

Beyond the standard checklist, ensure these maddy-local hypotheses (exact probes in maddy-task-guardian Step 4b):

• bleater-env is reachable by ubuntu (the #1 structural risk — if Forbidden, governance Layer 6 is unsolvable)
• All three drift controllers exist and are annotation/sidecar discoverable (kube-system, bleater-env, canary)
• Drift names are NOT leaked in the baseline config
• maddy-gitops-source is stale (replicas: 1, mode: direct)
• A no-op agent passes neither subscore

Phase 2.5 is auto-invoked. Treat any BLOCKING finding as a failure → auto-fix → re-loop. Skip Phase 2.5 only for a pure numeric tweak inside existing structure (e.g. a drift sleep value or a wait_consistent shift within ceiling). Any change that adds/removes resources, touches RBAC, or changes ALLOWED_NAMESPACES requires Phase 2.5.

If SSH to the VM fails, record the failure in the handoff message and proceed to Phase 3 with a noted risk — don't block the loop on environment issues.

Tell the user (verbatim, the ONLY user-facing message in the loop):

"Oracle passed at version N and live red-team is [clean | skipped: reason]. Please trigger a rollout batch for version N in the Horizon UI."

Phase 3 — Rollout analysis

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon rollouts pull --version NNN tasks/maddy-delivery-split-brain 2>&1

Poll every few minutes until ≥5 rollouts download. Read 2 transcripts (1 pass, 1 fail) before running the script — numbers tell you the difficulty; transcripts tell you why.

Analysis script

import json, glob
from collections import defaultdict

VERSION = "NNN"  # replace
TASK = "maddy-delivery-split-brain"
files = glob.glob(f"tasks/{TASK}/.rollouts/v{VERSION}/*.json")

scores = []
sub_vals = defaultdict(list)
joint = defaultdict(int)   # correlation tracker: (consistency, governance) -> count

for f in files:
    d = json.load(open(f))
    scores.append(d["score"])
    try:
        gr = json.loads(d.get("grade_result", "{}"))
        subs = gr.get("subscores", {})
        for k, v in subs.items():
            sub_vals[k].append(v)
        c = subs.get("delivery_consistency"); g = subs.get("observability_governance")
        if c is not None and g is not None:
            joint[(c, g)] += 1
    except Exception:
        pass

if not scores:
    print("No rollouts found — trigger an eval batch first")
else:
    n = len(scores); avg = sum(scores)/n
    pass_rate = sum(1 for s in scores if s >= 0.99)/n
    print(f"N={n}  avg={avg:.3f}  pass_rate={pass_rate:.1%}")
    for k, vs in sub_vals.items():
        vals = sorted(set(vs)); status = "DEAD" if len(vals)==1 else "varies"
        print(f"  {k}: {status} {vals}  mean={sum(vs)/len(vs):.2f}")
    print("  joint (consistency,governance):", dict(joint))
    half = sum(1 for s in scores if abs(s-0.5) < 0.01)
    correlated = (half == 0)  # no 0.5 rollouts => subscores move together
    if avg >= <TARGET_MEAN>:
        print("HIGH: avg >= <TARGET_MEAN> — invoke maddy-score-tuner")
    elif any(len(set(v))==1 for v in sub_vals.values()):
        print("DEADWEIGHT: a subscore is stuck — invoke maddy-score-tuner")
    elif correlated:
        print("CORRELATION COLLAPSE: no 0.5 rollouts — subscores move together — invoke maddy-score-tuner (Coupling/Decorrelate)")
    else:
        print("DONE: avg < <TARGET_MEAN>, both subscores vary, decorrelated")

Decision rules (Phase 3 outcomes)

Rollout result	Action
avg < <TARGET_MEAN>, both vary, some 0.5 rollouts	DONE
avg < <TARGET_MEAN> but a subscore deadweight	maddy-score-tuner — identify which is too easy/impossible
No 0.5 rollouts (perfect correlation)	maddy-score-tuner — add a decorrelating lever (don't pull a shared-drift lever)
observability_governance always 0.0	Suspect bleater-env reachability first (Dockerfile), then maddy-score-tuner
delivery_consistency always 0.0 (Oracle 1.0)	A drift source out-races the window → maddy-score-tuner (Consistency Specialist)
Either subscore always 1.0	maddy-score-tuner (discoverability too high)
Fewer than 5 rollouts	Poll again in 3 min

---

Decision rules (Phase 2 outcomes)

Oracle result	Action
1.0, passed: true, both subscores=1	Proceed to Phase 2.5
feedback: null, score: 0	Setup crash — maddy-validation-debugger Branch A (run No-Op to isolate)
score: 0, non-null feedback	Both failed — maddy-validation-debugger Branch E
0.5, delivery_consistency=0	maddy-validation-debugger Branch C
0.5, observability_governance=0	maddy-validation-debugger Branch D
Grader Python exception	maddy-validation-debugger Branch F

---

When to stop and ask for human review

Halt and summarize if any:

• 5 push cycles without measurable progress on a specific failure mode
• A proposed maddy-score-tuner fix would violate a hard constraint
• observability_governance stays at 0 despite solution.sh looking correct — invoke horizon-agentic-reviewer for a live bleater-env reachability probe before continuing
• avg stuck above 0.60 with no obvious lever left
• Two consecutive cycles worsened correlation (subscores moving together)

The summary should answer: which symptom is unresolved, which fixes were tried, current numbers, which specialist the user should look at next.

---

Skill dependency map

maddy-iteration-loop  (you are here — orchestrator)
├── maddy-task-guardian          Phase 0 pre-push, Phase 1 push, Phase 5 monitoring
├── maddy-validation-debugger    Phase 2 failure interpretation
├── maddy-score-tuner            Phase 3 score/variance/correlation diagnosis
└── horizon-agentic-reviewer     Phase 2.5 live red-team + any escalation needing live evidence

The orchestrator does not edit files itself — it delegates editing to the specialist that diagnosed the symptom, then applies through maddy-task-guardian's push flow. This diagnose → propose → approve → apply separation prevents blind edits.