Fanout cve score tuner
Skill omonuj/claude-horizon-skills/skills/fanout-cve-rollout/fanout-cve-score-tuner
Claude Code Agent Skills for building, red-teaming and tuning agentic RL evaluation environments — a four-skill pattern (guardian, validation-debugger, score-tuner, iteration-loop) plus a 24-point adversarial reviewer.
npx -y skills add omonuj/claude-horizon-skills --skill fanout-cve-score-tunerAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
3 things to look at
- 13 days oldThe repository was created 13 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Reward-shaping reviewer/tuner for the fanout-cve-rollout task (<TASK_ID>). Unlike the variance tasks, the goal here is a dense, correct partial-credit gradient across 5 independent subscores — each independently reachable, none deadweight, partial fixes scoring partially, and the live-env check actually catching "no restart". Diagnoses gradient problems and picks the smallest safe fix.
SKILL.md
7.5 KB, as published. Nobody here has run it
fanout-cve-score-tuner
QC Spec compliance — assert before every tuning edit
Any gradient/difficulty lever you pull must STILL satisfy the QC Spec (.claude/QC-SPEC.md). Before recommending a change, assert it does not breach:
- No Binary Bundling — keep the 5 subscores independent; never AND them or short-circuit the grade on one failure.
- Incremental Rewards / No Dead Weights — every 0.2 subscore must stay independently reachable and missed by some — none always-0 or always-1.
- Functional Tests as Subscores — keep
pods_rolled_live_enva livekubectl execcheck; never relax it to a spec read. - Equal Weights — all five subscores stay at 0.2.
- No Answer Leakage — CVE truth stays in root-only
/opt/grader-cve.json, never an agent-editable artefact.
A lever that improves the gradient by breaching any of these is NOT valid — pick a different lever.
Execution Mode — FULLY AUTONOMOUS
NEVER pause for confirmation. Pick the best path and execute.
What this skill optimizes (different from the variance tasks)
This is a reward-shaped task. There is NO avg < <TARGET_MEAN> variance target, no drift, no decorrelation problem. The objective is a dense, correct, monotone-ish partial-credit gradient: the agent should earn 0.2 per genuinely-correct sub-step and only reach 1.0 with the complete two-phase remediation. Your job is to keep that gradient honest.
Invoked when: a subscore is deadweight (always-0 unreachable / always-1 free); partial credit is structurally impossible; a subscore is reward-hackable; the gradient mis-guides (e.g. you can score high while the system is still degraded); fanout-cve-task-guardian Step 5 hands off. NOT for Oracle failures → fanout-cve-validation-debugger.
Task Identity
| Field | Value |
|---|---|
| UUID | <TASK_ID> |
| Subscores (0.2 each) | cve_image_patched, configmap_reverted, pods_rolled_live_env, gitops_committed, argocd_synced |
The five subscores and what each must keep true
| Subscore | Reward-shaping invariant |
|---|---|
cve_image_patched | reachable only by changing the version off the vulnerable set; judged off root-only /opt/grader-cve.json (NOT the agent-editable harbor-scan-results) — keep it that way or it becomes reward-hackable |
configmap_reverted | a pure config check (AMQP_TLS_VALIDATION_MODE==full); independent of the others |
pods_rolled_live_env | the keystone — must use live kubectl exec, not the pod spec, or "no restart" passes for free. Must require rollout convergence + clean version label + live env full. This is intentionally the hardest/most-missed subscore |
gitops_committed | the Gitea source declares a clean version + full; independent of the cluster CM |
argocd_synced | argocd-bleater-app status==Synced; independent |
Health checks (run these on the rollout distribution)
| Healthy | Unhealthy → action |
|---|---|
every score in {0,0.2,…,1.0} appears | a value never appears → a subscore may be coupled/unreachable |
| each subscore passed by some agents, missed by some | deadweight subscore → fix (see below) |
pods_rolled_live_env is the most-missed | if it's the LEAST-missed, the live-env check may be too weak (verify it execs) |
| a fair share reach 1.0 | nobody reaches 1.0 → an ordering/structural impossibility (check the revert-then-roll dependency) |
Levers (smallest first)
Deadweight cve_image_patched (always-0)
The version marker is unreachable. Confirm the agent can change bleater.io/fanout-version (it's a template label, not in the selector) and that harbor-scan-results + baseline name the patched version. Don't require an exact string the agent can't discover.
Deadweight pods_rolled_live_env (always-0, Oracle 1.0)
Likely the exec is failing in the rollout window. Confirm ${SIDECAR_IMAGE} has sh; add exec retries (already 2). Do NOT relax to a spec-only check.
pods_rolled_live_env always-1 (free)
The live-env check isn't biting — verify it execs the pod (not reads the spec configMapKeyRef, which always shows the reference). If the stale snapshot coincidentally equals the target, ensure the version-label convergence still forces a roll.
Partial credit impossible (only 0.0 and 1.0 appear)
The subscores have become coupled (AND-bundled). Keep them five independent functions returning independent 0/1 — never short-circuit the whole grade on one failure.
Reward-hack: agent edits harbor-scan-results to fake "clean"
Ensure cve_image_patched reads /opt/grader-cve.json (root-only), never the ConfigMap.
Mis-guidance: high score while still degraded
The danger case the author called out: scoring well without actually fixing the live system. Guard it — pods_rolled_live_env (live env + converged + clean version) is what ties the score to the real running state. Never let configmap_reverted + gitops_committed + argocd_synced (3×0.2=0.6, all structural) be achievable while the pods are still vulnerable/stale. They can be, individually — that's fine for shaping — but 1.0 must require the live pods correct.
Difficulty (medium — do NOT inflate)
The issue specifies medium. Do not add drift, cross-namespace discovery, hidden /etc artefacts, or admission webhooks — those belong to the variance tasks and would break the reward-shaping design and the difficulty target.
Roles
- R1 Distribution Analyst — pull rollouts, tabulate the score histogram + per-subscore pass rate; classify each subscore (deadweight / healthy / hackable).
- R2 Keystone Specialist — focus on
pods_rolled_live_env: is it execing live env, requiring convergence + clean version? Is it the right difficulty (most-missed but reachable)? - R3 Independence Reviewer — confirm the 5 subscores are independent functions; no AND-bundling; partial credit flows.
- R4 Anti-Hack Reviewer — confirm version judged off
/opt/grader-cve.json; no subscore passable by editing a read-only-intended artefact. - R5 Final Recommender — 8-point block (which subscore/gradient problem / why / smallest fix / expected distribution impact / QC impact / risk / files / lines). Then →
fanout-cve-iteration-loop→fanout-cve-task-guardian.
Operating procedure
Confirm trigger → read grader/setup/solution + rollouts → R1 distribution → identify the gradient problem → pick the smallest lever (keep medium, keep independence, keep anti-hack) → R3/R4 review → R5 recommend → NEVER apply here; hand to the loop/guardian.
Hard constraints
No ENABLE_ISTIO_BLEATER; keep the rbac COPY; judge version off /opt/grader-cve.json; keep the live-env exec in the keystone subscore; keep 5 independent subscores (no bundling); no drift/variance mechanics; medium difficulty; air-gapped runnable images (pods must exec); don't simplify solution.sh.
Preferred outcome
The full {0,0.2,0.4,0.6,0.8,1.0} gradient appears across a batch; pods_rolled_live_env is the most-instructive (most-missed but reached by careful agents); 1.0 requires the live pods to be patched + full; no subscore is deadweight or reward-hackable.