Audit with plans

Audit-with-plans

Audit-fix loop for projects using the planning-with-files skill. Repeatedly spawn independent fresh-context audit subagents that compare the code to .planning/<active>/{task_plan, findings, progress}.md and fix whichever side is wrong. Exit only when a fresh audit returns zero findings.

When to invoke

• The user asks for an audit, code review, or "is this project consistent?"
• A planning-with-files plan exists at .planning/<plan-id>/{task_plan, findings, progress}.md.
• You want to verify that the code does what the planning files claim, and that the planning files describe the code accurately.

If no planning-with-files plan exists, suggest the user start one (via the planning-with-files skill) before invoking this skill — the audit needs a baseline.

The loop

audit  →  fix  →  audit (fresh, no memory of prior pass)  →  fix  →  …  →  no findings  →  exit

Exit condition: an independent fresh-context audit subagent returns 0 CRITICAL / 0 HIGH / 0 MEDIUM / 0 LOW.

NOT exit conditions:

• "I think I fixed everything."
• "A subagent that I told my fix history to confirmed my fixes landed."
• "All HIGH/MEDIUM closed; LOW items remain."

Subagent prompt template

See templates/audit_subagent_prompt.md for the full parameterized prompt. The required structure is:

1. Role: "You are a code reviewer auditing <PROJECT_PATH>."
2. Project goal: one paragraph stating what the project is trying to achieve (auditor cross-checks against this).
3. Authoritative baseline: "Source of truth = .planning/<active-plan>/{task_plan, findings, progress}.md. Other .md files are docstring helpers; if they disagree with the planning files, the planning files win."
4. Section-level audit hooks (see below).
5. Two failure modes: (a) code does X but .planning says Y; (b) .planning describes a workflow / artifact / value that the code is missing / broken / out of sync. Either side can be wrong.
6. Execution requirement: "For every documented entry point, execute with the documented invocation, check exit code, compare stdout/stderr/file outputs to the .planning description."
7. Artifact verification: "For every on-disk artifact .planning/*.md cites, verify the file exists; if a specific number is cited, grep for it in the file; if script-generated, regenerate + diff against the on-disk version where practical."
8. Safety bounds: "DO NOT modify any files. DO NOT submit SLURM jobs / batch jobs. DO NOT SSH anywhere. Report only."
9. Independence: "You have no prior context on this project. Do not assume any audit has run before. Do not look for 'things prior passes might have missed'. Start cold from the three planning files and work outward."

Required: NEVER include in the subagent prompt

• A list of prior findings.
• A list of what was just fixed.
• "Pass-N closed X; verify each landed."
• "Don't re-flag closed items."
• Any narrative from your prior fix history.

These turn the auditor into a verifier of your fixes. The verifier confirms what you told it is fixed and misses what you introduced, rationalized away, or never noticed. Each pass must be cold-start.

Section-level audit hooks (planning-with-files structure)

Anchor the subagent's checks to the canonical sections of the three planning files:

File	Section	What to verify
task_plan.md	Goal	Code does what the goal says
task_plan.md	Current Phase	Code state matches the named current phase
task_plan.md	Phases (status complete)	Artifacts the phase claims to produce exist
task_plan.md	Decisions Made (table)	Every decision reflected in code
task_plan.md	Open Blockers	Every row in the described state
task_plan.md	Errors Encountered	Every row's resolution applied (or explicitly still open)
findings.md	Technical Decisions + Resources	Every cited file path exists; every cited numeric value (RMSE, density, monopole, ε, R_min, etc.) appears in the referenced artifact at the cited precision
progress.md	Files created/modified lists	Every cited path exists
progress.md	5-Question Reboot Check	Answers reflect CURRENT state, not a snapshot from an earlier session
progress.md	Test Results	Every row's "Actual" reproduces on a fresh run
meta / tooling	.planning/.active_plan + self-audit script	The project's self-audit script (audit_planning.py) reads .active_plan and dispatches the matching check set — NOT hard-coded to a stale plan. Confirm which plan's checks actually fire.
.md ↔ .md	across task_plan / findings / progress	Internal consistency: Current-Phase vs phase statuses; "decision pending" vs "made"; every shared numeric value identical across all three files

Bidirectional fix policy

For each discrepancy:

• If the code does X and .planning says Y, and .planning is RIGHT → fix the code.
• If the code does X and .planning says Y, and the code has legitimately evolved (algorithm changed, cutoff tightened, path renamed) → fix the .planning file.

Stale documentation is as broken as stale code. Either direction is valid; the goal is convergence between code and .md, not one-way fixing of code.

Fix every reported issue

When an audit returns findings:

• Fix all of them (CRITICAL through LOW).
• The audit subagent may explicitly mark some as "no fix required" / "informational" / "open by design pending user decision" — those don't count against the severity counts but should be acknowledged in the exit report.
• Do NOT punt items to "next session" without explicit user approval.
• Do NOT batch fixes across multiple audit passes; close each pass's findings before spawning the next.

Anti-patterns

Anti-pattern	Symptom	Correction
Verification bias	Subagent prompt includes "Pass-N closed X; verify each + don't re-flag"	Spawn each audit with no prior context — role + goal + baseline + execution requirement only
Static reading	Subagents grep text but never execute scripts	Mandate execution of every documented entry point with trivial inputs
Confidence over-calibration	Declaring "converged" on subagent's "0 findings" without independent verification	Treat subagent results as hypotheses; cross-check the most critical entry-point yourself
Cookie-cutter "no action" deferrals	LOW/MEDIUM punted to "next session"	Fix every reported issue (LOW too) and re-audit until clean
Reading-vs-behavior gap	Caught text drift; missed CWD bugs, silent failures, validator false-OKs	Prompt phases for: (a) library docstring vs production workflow, (b) silent-failure grep for except Exception + missing exit codes, (c) audit-script positive/negative path inspection
One-direction fixing	Always changing code to match .md	Recognize when code is right and .md is stale; fix the .md side
Iteration short-circuit	Spawning a single audit subagent and declaring done	The loop runs until a FRESH audit returns 0/0/0/0 — typically 5-15 passes for a non-trivial project
Assembled-path blindness	Path-update sweep only greps for literal old prefixes; scripts that assemble paths from variables (e.g. OTF / "01_qm_opt/sp.fchk" where OTF is a Path constant) silently break	Audit subagent prompt must require: (a) bash -n / python3 -m py_compile parse-check, (b) actual execution with trivial input, (c) running any test suite (pytest/unittest) found in the project
Template-vs-instance drift	An on-disk template is upgraded with new safety checks; pre-existing copies of that template in working dirs still have the OLD version	Audit must diff every dynamic.sh / minimization.sh / build.sh in working dirs against the canonical template; flag any divergence
READMEs overstating completion	A README says "Phase N complete" but the cited observable (e.g. ΔH^vap = X kJ/mol) is missing from any artifact	Audit must verify every "complete" claim has a non-empty, current-code-produced artifact backing it. .CRASHED_* archives DO NOT count as a successful run; a dynamics.log without Performance: marker DOES NOT count as a complete trajectory
audit_planning.py false confidence	Project's own self-audit script checks file existence only; entry-point breakage invisible to it	The project's audit_planning.py must include _audit_entry_points_runnable (parse-check every documented script) + _audit_phase_completion (verify observables exist, not just files)
Self-audit script targets the WRONG plan	The project's self-audit script (audit_planning.py) is hard-coded to one plan id; .planning/.active_plan has since switched to a different plan. The script keeps printing "27 OK" — but it is auditing the stale plan's artifacts, not the active one. Subagents read those OKs as evidence the active plan is consistent. This is the most dangerous failure mode: false confidence at the tooling layer.	EVERY pass must (1) read .planning/.active_plan, (2) confirm the self-audit script resolves and dispatches on that same id — open audit_planning.py and check it reads .active_plan rather than a hard-coded constant. If the script audits a different plan than the active one, that is an automatic HIGH finding. The fix is to make the script resolve_active_plan() + dispatch per-plan check sets, gating each plan's checks behind its id.
.md ↔ .md internal contradiction	Audit checks only code↔.md and misses contradictions between the three planning files: task_plan.md "Current Phase" disagrees with the phase-status checkboxes; findings.md says a decision is "pending" while task_plan.md Decisions Made records it as made; a numeric value (production λ, monopole, density) differs between task_plan / findings / progress	Add an explicit .md↔.md hook: cross-check Current-Phase vs phase statuses; every "decision pending/made" claim across the three files; every shared numeric value (grep the value across all three). Planning-file self-consistency is part of convergence.
Auditor's own fixes leave dangling cross-refs	The agent fixes a value in task_plan.md but the old value still lives in findings.md/progress.md. The next homogeneous pass shares the agent's blind spot and "converges" without catching it.	After changing ANY value in a planning file, grep the old value across all three planning files (and READMEs / code comments) and update every hit. Treat your own edits as a new source of drift to re-audit.
Homogeneous-prompt fixed point	Every pass uses the same prompt template anchored on the same hooks, so the loop converges to a fixed point that still contains a systematic blind spot the template never looks at	Vary the lens across passes (or always include a "meta/tooling" hook: does the self-audit tool target the active plan? are the three planning files mutually consistent?). A clean 0/0/0/0 from a homogeneous prompt is weaker evidence than one finding from a differently-framed pass.

Termination report template

After a plan switch (.planning/.active_plan changed)

Switching the active plan is a major event on par with a restructure — it silently invalidates all plan-scoped tooling. When .active_plan changes (or when you start an audit and find the project has multiple plans):

1. Verify the self-audit script (audit_planning.py or equivalent) resolves .active_plan and dispatches the matching check set. A script hard-coded to the previous plan will print OKs about the wrong artifacts. This is an automatic HIGH finding until fixed.
2. Verify every plan-scoped tool / path constant points at the active plan, not the old one.
3. Re-run the self-audit and read WHICH plan's checks fired — not just the OK/FAIL counts. If you see checks naming the inactive plan's artifacts, the dispatch is broken.
4. The exit condition (0/0/0/0) is only meaningful once the self-audit is confirmed to be auditing the active plan.

After a major restructure (paths moved, files relocated)

Path-update sweeps with grep+sed are necessary but NOT sufficient. They catch literal-string references. They miss:

1. Assembled paths — OTF / "01_qm_opt/sp.fchk" where OTF is a Path variable; the prefix is in the variable, the suffix in the assembled string.
2. Test expectations — tests usually use Path(__file__).parent.parent / "templates"; restructure breaks them silently until pytest runs.
3. Template inheritance gap — the canonical template was updated; existing on-disk copies of that template (in working dirs, T-folders, etc.) were not re-templated.
4. README claims — "Phase complete" was true before the move but now references files at stale paths or observables that haven't been measured.

Mandatory post-restructure audit pass:

• bash -n on every shell script under the new tree.
• python3 -m py_compile on every Python script.
• Run every documented entry-point script with a trivial invocation; check exit code.
• Run the project's test suite (pytest etc.).
• Regenerate every cited analysis output; diff against on-disk version.
• Diff every working-dir copy of dynamic.sh / minimization.sh / build.sh against the canonical templates.
• For every phase N complete claim, verify the artifact exists, is non-empty, and was produced by the current code.

Termination report template

When the loop converges, report:

## Loop converged.

Fresh independent audit returned 0 CRITICAL / 0 HIGH / 0 MEDIUM / 0 LOW.

| Check | Status |
|---|---|
| Every cited file path in planning | ✓ exists |
| Every cited numeric value | ✓ matches artifact to documented precision |
| Every documented entry point | ✓ exit 0 |
| <add other categories specific to the project>
| ✓ |

One explicitly open item (documented, not an audit failure): <if any>

The remaining work is purely <physics / business logic / etc.>.

If items remain documented-but-unresolved (e.g. an explicit Open Blocker awaiting user decision), name them in the termination report so the user can see what's deferred by intent vs what's deferred by oversight.

Anti-patterns observed in the source session (2026-05-26)

This skill was distilled from a 12+ pass audit session where the agent (me) repeatedly claimed convergence and was repeatedly wrong. The user had to teach the correct loop explicitly. The behaviors codified here all map to mistakes that session made:

• Declared "converged" after passes 1, 3, 5, 7 — all wrong; each time the user produced new bugs in minutes.
• Every subagent prompt seeded the auditor with prior findings — they verified fixes, didn't find new bugs.
• Subagents only read code; they never ran bash submit_solution_build.sh … from the documented directory, which would have caught a CWD-vs-path bug in seconds.
• Only fixed code to match .md; never recognized the .md could be the stale side.

The user's direct corrections — "the loop is: audit, fix, audit again as fresh, not previous memory or context, find questions?, fix, audit with no previous memory or context, no findings, clean and report exit the loop" — and "treat the audit as a code reviewer trying to read/find the discrepancy between the code and the .md, if we have the correct .md, then we need to fix the code, if we have the code updated, we should update the .md as well" — are the canonical statement of this skill.

Related skills / memories

• planning-with-files — prerequisite. audit-with-plans audits the artifacts that planning-with-files produces.
• Memory rules consolidated into this skill (project-scoped, may also exist in your memory dir):
  • feedback_audit_loop_must_continue.md — fix every reported issue.
  • feedback_audit_must_test_behaviors.md — check provenance, silent failure, validator bugs.
  • feedback_audits_must_execute.md — execute entry points; regenerate outputs; grep docs for numeric claims.
  • feedback_independent_audit_loop.md — each pass context-free; bidirectional fix.