Mtls iteration loop

mtls-iteration-loop

Execution Mode — FULLY AUTONOMOUS

NEVER call ask_user or pause for confirmation at any point. Run every phase, every bash command, and every fix without asking permission. If a decision has two valid paths, pick the safer one and proceed. The user triggered this skill precisely to avoid being prompted — do not interrupt them.

Goal

Drive the variation-fanout-pipeline-break task from "needs work" to "shippable" without human intervention except for triggering the rollout batch in the Horizon UI. The loop is done when all exit conditions hold; until then, every cycle picks the right specialist skill (validation-debugger, score-tuner, horizon-agentic-reviewer) for the symptom and applies the smallest safe fix.

This is the front-door skill for "iterate this task to a passing state." The other three mtls-* skills are pure capabilities; this skill orchestrates them.

Trigger

Use this skill when asked to:

• "Iterate the mtls task until it passes"
• "Push and tune until avg < <TARGET_MEAN>"
• "Run the full loop on variation-fanout-pipeline-break"
• Anything implying repeated push → validate → analyze → fix cycles

For single-step requests (just push, just analyze rollouts, just debug a validation), invoke the relevant specialist skill directly.

Task Identity

Field	Value
Task UUID	<TASK_ID>
Task slug	variation-fanout-pipeline-break
Local path	tasks/variation-fanout-pipeline-break/
Horizon root	/Users/mac/Documents/tasks
Venv	source /Users/mac/Documents/tasks/horizon_env/bin/activate

---

Who does what

The agent (you) executes every step except 3c. Never tell the user to run push, validate, or any CLI command — those are yours.

Step	Who	What
Edit files (setup.sh, grader.py, solution.sh, task.yaml, Dockerfile)	Agent	via Edit/Write tools
Pre-push syntax + anatomy + quality checks	Agent	via Bash tool
Push task (horizon tasks push)	Agent	via Bash tool
Oracle validation (horizon tasks validate -a oracle --wait)	Agent	via Bash tool
Live red-team on the live container	Agent	invokes horizon-agentic-reviewer
Pull rollouts (horizon rollouts pull)	Agent	via Bash tool
Analyze rollouts (variance, deadweight, pass-rate)	Agent	via Bash tool, script below
Diagnose & propose tuning fix	Agent	invokes mtls-score-tuner
Diagnose & propose validation fix	Agent	invokes mtls-validation-debugger
Trigger the eval batch in the Horizon UI	User	the only manual step

---

Exit conditions

The loop is DONE when all of these hold simultaneously on the same version:

1. Oracle validation: passed: true, score: 1.0, both subscores =1
2. Live red-team via horizon-agentic-reviewer: no BLOCKING findings (checks 2, 11, 13, 16, 17, 21)
3. Rollout avg: < <TARGET_MEAN> (Nebula creator workflow requirement)
4. mtls_handshake: varies across the rollout batch — both 0 and 1 appear
5. trust_governance: varies across the rollout batch — both 0 and 1 appear
6. Local quality check: 18/20 (or 15+/20 with only the four known noise checks failing — see mtls-task-guardian Step 2)

Anything short of all six → loop continues.

Loop limit

Do not exceed 5 push cycles without human review. After 5 cycles with no measurable progress on a specific failure mode, stop and write a summary of what was tried and what is still failing. The user is faster than 5 more cycles at that point.

No-Op default

Do NOT run No-Op validation as part of the loop. Oracle only. Oracle exercises the same setup.sh path No-Op does (Oracle = setup.sh + solution.sh + grader). If Oracle scores 1.0 cleanly with both subscores recovered, setup.sh ran fine — No-Op adds no signal.

Run No-Op manually only when Oracle returns feedback: null or score: 0 with no grader detail — that pattern indicates a setup crash, and No-Op isolates whether it's setup or solution.sh interference.

---

The loop

START
  │
  ▼
┌─ PHASE 0: Pre-push checks ──────────────────────────────────────┐
│  • Dockerfile invariants (mtls-task-guardian Step 0)            │
│  • bash -n setup.sh && bash -n solution.sh && py_compile grader │
│  • horizon check-anatomy (must pass clean)                      │
│  • horizon check-quality (expect 18/20; accept 15+/20 if only   │
│    the four known noise checks fail)                            │
│  If anything else fails → fix → re-run PHASE 0                  │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 1: Push ─────────────────────────────────────────────────┐
│  horizon tasks push tasks/variation-fanout-pipeline-break   │
│  → record version number NNN                                    │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 2: Oracle validation ────────────────────────────────────┐
│  horizon tasks validate -m hosted -a oracle --wait              │
│  → Oracle 1.0 + both subscores=1 → PHASE 2.5                    │
│  → anything else → invoke mtls-validation-debugger → fix →      │
│    back to PHASE 0                                              │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 2.5: Live red-team (mtls-task-guardian Step 4b) ─────────┐
│  Why: 10 min here saves a 60 min rollout cycle.                 │
│  When: every push where setup.sh, grader.py, or task.yaml       │
│        changed. Skip only for pure grader-timing tuning with    │
│        no structural change.                                    │
│  How: invoke horizon-agentic-reviewer on task UUID              │
│       <TASK_ID>.                     │
│  → no BLOCKING findings → ask user to trigger rollout batch     │
│  → BLOCKING finding → fix → back to PHASE 0                     │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 3: Rollout analysis ─────────────────────────────────────┐
│  horizon rollouts pull --version NNN  (re-run until ≥5 rollouts)│
│  Read at least 2 transcripts (one pass, one fail) BEFORE tuning │
│  Run analysis script (below)                                    │
│  → avg < <TARGET_MEAN> AND both subscores vary → DONE                    │
│  → anything else → invoke mtls-score-tuner → fix → PHASE 0      │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
DONE

---

Phase 0 — Pre-push commands

cd /Users/mac/Documents/tasks

# 1. Dockerfile invariants (see mtls-task-guardian Step 0)
echo "=== Dockerfile ===" && cat tasks/variation-fanout-pipeline-break/Dockerfile
#   Must NOT contain: ENABLE_ISTIO_BLEATER
#   Must contain:     ALLOWED_NAMESPACES="kube-system"
#   Must contain:     COPY data/ubuntu-user-rbac.yaml

# 2. Syntax
bash -n tasks/variation-fanout-pipeline-break/setup.sh    && echo "setup.sh OK"
bash -n tasks/variation-fanout-pipeline-break/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/variation-fanout-pipeline-break/grader.py && echo "grader.py OK"

# 3. Anatomy + quality
source horizon_env/bin/activate
horizon check-anatomy tasks/variation-fanout-pipeline-break 2>&1
horizon check-quality tasks/variation-fanout-pipeline-break 2>&1 | tail -25

Phase 1 — Push

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/variation-fanout-pipeline-break 2>&1
# Record: "✓ New version pushed successfully! Version: NNN"

Phase 2 — Oracle validation

Do NOT rely on --wait alone. The horizon tasks validate --wait CLI uses TTY progress spinners that don't flush properly when running non-interactively in background. The completion message gets buffered and never written to the output file until the parent exits, so the foreground "wait" can hang for 30+ minutes after the hosted run has actually completed. Use a poll-via-validate-logs pattern instead.

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

# 1. Trigger validation (no --wait, returns immediately)
horizon tasks validate -m hosted -a oracle tasks/variation-fanout-pipeline-break 2>&1
# Capture the Build ID from output: "Build ID: val-6a7cfb6a-<timestamp>"
# (you can also derive it later from the .validation/ subdirectory name)
BUILD_ID="val-6a7cfb6a-<TIMESTAMP_FROM_OUTPUT>"
RESULT_PATH="tasks/variation-fanout-pipeline-break/.validation/${BUILD_ID}/result.json"

# 2. Poll every 60s by running validate-logs.  CRITICAL: the result.json is
#    created early in the build lifecycle with `status: "running"` and updated
#    later when the run completes.  Do NOT exit the poll on file existence
#    alone — check that status != "running" (i.e. "passed" / "failed").
for attempt in $(seq 1 30); do
  sleep 60
  horizon tasks validate-logs -a oracle tasks/variation-fanout-pipeline-break >/dev/null 2>&1
  STATUS=$(python3 -c "import json; print(json.load(open('$RESULT_PATH')).get('status', 'unknown'))" 2>/dev/null)
  if [ "$STATUS" != "running" ] && [ -n "$STATUS" ]; then
    echo "Oracle completed after ${attempt} minute(s), status=$STATUS"
    cat "$RESULT_PATH" | python3 -m json.tool
    break
  fi
  echo "[poll ${attempt}/30] status=$STATUS"
done

Pass criterion: score: 1.0, passed: true, both mtls_handshake=1 and trust_governance=1.

Operational note: when running this via a background bash task, use run_in_background: true and let the harness notify on completion — do not foreground-wait inside a 2-minute Bash tool call.

Anything else → invoke mtls-validation-debugger (do NOT edit files directly first).

Phase 2.5 — Live red-team

Invoke horizon-agentic-reviewer against task UUID <TASK_ID>. The reviewer will:

1. Spin up a live container on the Nebula Aurora VM (ssh nebula-vm)
2. Run horizon setup against the latest version
3. Execute the task's setup.sh manually (horizon setup does NOT do this automatically — see horizon-agentic-reviewer.md Step 3.5)
4. Probe the container as the agent user (ubuntu for this task — it's on the nebula-devops image lineage)
5. Run the 24-point checklist

In addition to the standard checklist, ensure these task-local hypotheses are checked (see mtls-task-guardian Step 4b for the exact probes):

• All four drift sources are discoverable to an ubuntu-perspective audit
• Strategy A (bleater-profile-cache-sync) is hidden — not named in baseline config
• HPA pinner on bleater-cert-reaper is in place
• A no-op agent cannot pass either subscore

Phase 2.5 is auto-invoked. Do not ask the user for approval — just run it. The reviewer writes findings.json; read it programmatically. Treat any BLOCKING finding (checks 2, 11, 13, 16, 17, 21 — see horizon-agentic-reviewer's coverage map) as a failure → auto-fix and re-loop. Non-blocking findings can be deferred but should be noted in the rollout-handoff message.

Skip Phase 2.5 only when the change is purely a numeric tweak inside an existing structure (e.g. changing time.sleep(45) to time.sleep(30), or wait_consistent parameter shifts within ceiling). Any change that adds/removes resources, modifies metadata (finalizers, labels, owner references), or touches RBAC requires Phase 2.5.

If SSH to the Nebula Aurora VM fails (auth / connectivity), record the failure mode in the rollout-handoff message and proceed to Phase 3 with a noted risk — do NOT block the loop on environment issues. The rollout itself is the next-best-evidence ground truth.

Only after Phase 2.5 is clean (or skipped per the rules above) do you proceed to Phase 3.

Tell the user (verbatim, the ONLY user-facing message in the loop):

"Oracle passed at version N and live red-team is [clean | skipped: reason]. Please trigger a rollout batch for version N in the Horizon UI."

Phase 3 — Rollout analysis

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

# Replace NNN with the version pushed in Phase 1
horizon rollouts pull --version NNN \
  tasks/variation-fanout-pipeline-break 2>&1

Poll every few minutes until at least 5 rollouts have downloaded. Then read 2 transcripts before running the script — one pass, one fail. The numbers tell you whether the task is at the right difficulty; the transcripts tell you why agents are succeeding or failing.

Analysis script

Save as /tmp/analyze_rollouts.py or paste into a Python REPL. Replace NNN with the current version.

import json, glob
from collections import defaultdict

VERSION = "NNN"  # replace
TASK = "variation-fanout-pipeline-break"
files = glob.glob(f"tasks/{TASK}/.rollouts/v{VERSION}/*.json")

scores = []
sub_vals = defaultdict(list)

for f in files:
    d = json.load(open(f))
    scores.append(d["score"])
    try:
        gr = json.loads(d.get("grade_result", "{}"))
        for k, v in gr.get("subscores", {}).items():
            sub_vals[k].append(v)
    except Exception:
        pass

if not scores:
    print("No rollouts found — trigger an eval batch first")
else:
    n = len(scores)
    avg = sum(scores) / n
    pass_rate = sum(1 for s in scores if s >= 0.99) / n
    print(f"N={n}  avg={avg:.3f}  pass_rate={pass_rate:.1%}")
    print()
    for k, vs in sub_vals.items():
        vals = sorted(set(vs))
        status = "DEAD" if len(vals) == 1 else "varies"
        mean = sum(vs) / len(vs)
        print(f"  {k}: {status} {vals}  mean={mean:.2f}")
    print()
    if avg >= <TARGET_MEAN>:
        print("HIGH: avg >= <TARGET_MEAN> — invoke mtls-score-tuner")
    elif any(len(set(v)) == 1 for v in sub_vals.values()):
        print("DEADWEIGHT: a subscore is stuck — invoke mtls-score-tuner")
    else:
        print("DONE: avg < <TARGET_MEAN> and both subscores vary")

Decision rules (Phase 3 outcomes)

Rollout result	Action
avg < <TARGET_MEAN> AND both subscores vary	DONE — exit loop
avg < <TARGET_MEAN> BUT a subscore is deadweight	Invoke mtls-score-tuner — identify which subscore is too easy/impossible
mtls_handshake always 1.0	Invoke mtls-score-tuner (Handshake Specialist role)
trust_governance always 1.0	Accepted if avg < <TARGET_MEAN>; otherwise invoke mtls-score-tuner (Governance Specialist role)
trust_governance always 0.0	Likely structural — check ALLOWED_NAMESPACES in Dockerfile and kube-system RBAC in setup.sh before invoking mtls-score-tuner
mtls_handshake always 0.0	Re-verify Oracle first via mtls-validation-debugger. If Oracle is 1.0, drift loop is too fast for grader window — invoke mtls-score-tuner
Fewer than 5 rollouts pulled	Poll again in 3 min — minimum 5 needed for meaningful stats

---

Decision rules (Phase 2 outcomes)

Oracle result	Action
score: 1.0, passed: true, both subscores=1	Proceed to Phase 2.5
score: 1.0 but passed: false	Grader threshold drift — should not happen with current grader; invoke mtls-validation-debugger
feedback: null, score: 0	Setup crash — invoke mtls-validation-debugger Branch A
score: 0, non-null feedback	Both subscores failed — invoke mtls-validation-debugger Branch E
score: 0.5, mtls_handshake=0, trust_governance=1	Invoke mtls-validation-debugger Branch C
score: 0.5, mtls_handshake=1, trust_governance=0	Invoke mtls-validation-debugger Branch D
Grader Python exception	Invoke mtls-validation-debugger Branch F

---

When to stop and ask for human review

Halt the loop and write a summary if any of these:

• 5 push cycles completed without measurable progress on a specific failure mode
• A proposed fix from mtls-score-tuner would violate a hard constraint (see mtls-task-guardian Hard constraints table)
• Oracle consistently returns score: 0.5 on trust_governance despite solution.sh appearing correct — invoke horizon-agentic-reviewer for a live probe before continuing
• Rollout avg is stuck above 0.60 despite no obvious tuning lever remaining
• Two consecutive cycles introduced coupling (changing handshake moved governance, or vice versa)

The summary should answer: which symptom is unresolved, which fixes were tried, what the current numbers are, and which specialist role the user should look at next.

---

Skill dependency map

mtls-iteration-loop  (you are here — orchestrator)
├── mtls-task-guardian          Phase 0 pre-push checks, Phase 1 push, Phase 4 monitoring
├── mtls-validation-debugger    Phase 2 failure interpretation
├── mtls-score-tuner            Phase 3 score/variance diagnosis and lever selection
└── horizon-agentic-reviewer    Phase 2.5 live red-team and any escalation requiring live evidence

The orchestrator does not edit files itself — it delegates editing to the specialist that diagnosed the symptom. The specialist proposes the edit; the orchestrator (this skill) applies it through mtls-task-guardian's push flow. This separation prevents the "blind edit" failure mode that wasted cycles in earlier versions.