Rules from coding agent failures

Rules from Coding Agent Failures

Run the failure-to-rule loop autonomously: capture agent failures, triage them into clusters, promote recurring patterns into the smallest durable fix, verify closeout, prune stale rules. State assumptions and evidence in the output; ask only on genuine blockers.

Core invariants

• The ≥3 promotion floor (W1). Recording is cheap: one observation with a non-trivial "What to do differently" line is worth logging. Promotion — turning entries into a rule, hook, or CI gate — requires three or more entries describing the same gap; fewer overfits to plausible boilerplate that measurably hurts agents. Never filter single observations at recording time — that starves promotion. Details: references/empirical-warnings-w1.md.
• A log is not a loop. Entries are telemetry until they change the next run — a rule, hook, gate, eval, test, or docs fix with a verification rule.
• Gates guarantee, prose guides. Hooks and CI gates enforce near 100%; instruction-file prose holds ~70%. Prefer hook > CI gate > prose.
• Rules expire. Record the limitation each promoted rule assumes; when the gap stops recurring, propose removal. Promotion without pruning accretes stale, contradictory rules.

Workflow

1. Capture. Scaffold docs/reflection-log/ (README index plus _template.md; per-entry frontmatter: date, harness, sub-surface, severity, status, related) and append a pointer to it into an existing always-loaded surface — the repo README or instructions file, edited in place, not a new file. The scaffolded README must state the recording-bar / promotion-bar split. Shape, gotchas, staging: references/reflection-log.md.
2. Triage. Cluster entries by frontmatter (grep -l 'sub-surface: X' docs/reflection-log/[0-9]*.md). Three or more same-gap entries: promote candidate; two: watch; one: record only. Name each cluster's closure surface and one verification rule.
3. Promote. For each at-floor cluster, propose the smallest durable change and cite the entry filenames that justify it. State the evidence for and against the promotion in the output instead of asking. For a hook, static check, or CI gate, apply the variant matrix and regression-fixture requirements in references/gate-hardening.md.
4. Closeout. Re-run each cluster's verification rule. Mark entries resolved only with concrete evidence (command output, fixture run, CI log); otherwise leave open, labeled needs_evidence.
5. Prune. Re-check previously promoted rules; propose retiring any whose underlying failure no longer reproduces.

Stop conditions

• No log and no observed failures: stop after capture — nothing to triage or promote yet.
• Cluster below the floor: record or watch; a draft closure plan is the only permitted promote output.
• Told this is the wrong direction: restate the corrected goal and re-plan; do not patch forward.