Audit loop

Audit loop

A draft answer is verified by an independent re-solve plus a cross-method probe, not by critique. The subagents never see the draft; they work the original problem from independent angles. The three independent results (draft, re-solve, probe) are then mechanically compared.

Why re-solve, not critique

Empirical research on LLM-as-judge shows critique-of-draft has structural failure modes that cannot be patched by prompt engineering:

• Refinement-aware bias — same content scored higher when framed as "revised."
• CoT trust — judges believe shown reasoning traces as ground truth.
• Sycophancy — multi-turn argument produces 3× more flips.
• Answer wavering — multi-round critique echo-chambers rather than converging.
• Self-preference / perplexity bias — same-family model under-flags errors typical of its own training distribution.

See References below for citations. Recomputation/replication is the load-bearing mechanism in every mature human audit domain (reperformance > inquiry in financial audit; replication > peer review; kernel-check > read-the-proof). Step-checking inherits the auditee's blind spots.

Triage gate

Claim type	Route
Quantitative / external-fact claim	→ use the user's red-team-process memory (measurement-first + PROVENANCE BLOCK). Stop.
Fully covered by tests, or cheap to test exhaustively	→ run the tests. Skip audit-loop.
Partially testable — some empirical surface but invariants, adversarial inputs, or edge cases lie outside test coverage	→ run available tests AND apply audit-loop to the uncovered surface. Record both in the audit line.
Pure reasoning, no empirical surface	→ full audit-loop.
Lookup, opinion, or trivial	→ skip.

The middle row matters. Most algorithm-correctness claims sit here: you can test typical inputs, but invariants and adversarial behavior aren't covered. Don't let the triage push such claims fully into either column.

Protocol

Step 1 — Characterize (internal scratch, do not display)

• CLAIM: one sentence stating what the answer asserts.
• FALSIFICATION SHAPE: what would count as proof this is wrong? Pick the most concrete form:
  • A specific input producing the wrong output
  • A counterexample to a property
  • An independent recomputation yielding a different number
  • A construction that breaks an invariant
  • An attack that breaches a safety claim
• If the falsification shape is unclear after honest thought, note warnings=falsification-unclear; protocol value is degraded.

Step 2 — Spawn 1: Independent re-solve

Spawn one subagent via the Agent tool with subagent_type=general-purpose. Prompt structure:

Solve the following problem. Return your answer in this format: <format>.

<PROBLEM>
{user's original problem statement, transcribed verbatim — DO NOT paraphrase}
</PROBLEM>

These are construction rules for YOU (the main agent), not text to put in the subagent's prompt. The subagent should see only the minimal "solve" framing above. Explicit mentions of forbidden entities (the draft, the audit context, the verification framing) trigger priming failure: naming X is the dominant way "don't do X" leaks X into the output — ~87.5% of negation violations come from explicit mention of the forbidden entity (Rana 2026, https://arxiv.org/pdf/2601.08070). The fix is structural omission, not negative instruction: don't include those entities in the prompt at all, rather than instructing the model not to use them.

Construction rules:

• Transcribe verbatim. Paraphrasing smuggles in main-agent framing and loses constraints. Use the user's exact words. For multi-turn conversations where no single user message states the full problem, see Multi-turn handling below.
• Do NOT include the draft, your reasoning, or any framing about audit/verification in the prompt.
• Do NOT use "audit / critique / find errors / verify" wording in the prompt. Just "solve." Adversarial framing past threshold produces hallucinated findings rather than calibrated skepticism.
• Specify the output format so comparison is mechanical (see Equivalence semantics below).

Step 3 — Spawn 2: Cross-method probe

Spawn 2 is always a probe via different method, not a third re-solve. This is the empirically well-supported choice:

• BoN-MAV (https://arxiv.org/abs/2502.20379, Table 4): diverse verifiers consistently outperform repeated queries to the best single verifier.
• Diversity of Thought (https://arxiv.org/abs/2310.07088): +10-30 percentage point gains over same-prompt self-consistency on Blocksworld, AQUA-RAT, GSM8K.
• Mechanism: same-family models agree on wrong answers ~60% of the time when both are wrong, vs ~33% by chance (Correlated Errors, https://arxiv.org/abs/2506.07962). A same-method re-solve inherits that coupling.

A probe via the FALSIFICATION SHAPE breaks the coupling by changing the angle of attack. (Caveat: whether the probe is most valuable on agreement (catching correlated agreement) or on disagreement (resolving conflict) is a design hypothesis — the literature supports method diversity, but the specific allocation has no head-to-head study.)

Falsification shape	Probe task
Specific input → wrong output	"Trace this algorithm/computation on inputs [concrete edge cases]; return outputs."
Counterexample to a property	"Find an input that violates property Y, or after honest effort report 'no case found.'"
Independent recomputation	"Compute X using [different decomposition / different base rate / Fermi-style buildup]; return result."
Construction breaking invariant	"Construct an input that would break invariant Z."
Attack breaching safety claim	"Construct an attack that violates safety property W, or report 'no attack found' after honest effort."
No clean probe (mostly design problems)	See Design fallback below.

Probe subagent gets concrete inputs but never the draft.

Step 4 — Three-way comparison

Compare draft, Spawn 1 (re-solve), and Spawn 2 (probe).

Pattern	Action
All three agree	Strong evidence. Ship.
Draft + re-solve agree; probe finds concrete counterexample	Probe is independent constructive evidence. Revise per probe.
Draft + probe agree; re-solve disagrees	Re-solve was an outlier; ship draft.
Re-solve + probe agree; draft disagrees	Strong signal draft is wrong. Revise to re-solve / per probe.
All three disagree	Admit uncertainty. Report all three to user.

This is "third independent angle, not vote" — the probe is constructive evidence (counterexample / trace / alternative computation), not an averaged opinion.

Equivalence semantics

String-level comparison is too brittle (0.5 vs 1/2 vs ½; lists in different order; algorithms phrased differently but with identical I/O). The main agent applies documented equivalence normalizations before declaring agreement:

• Numbers: canonical decimal at a stated precision; equivalent fractions; equivalent units.
• Sets / unordered lists: order-insensitive.
• Ordered sequences: order-sensitive (state which it is in the requested output format).
• Algorithms / code: identical I/O behavior on the probe inputs counts as equivalent, regardless of phrasing.
• Natural-language claims: equivalence requires exact semantic match on the load-bearing parts; if it requires interpretation, that's disagreement.

When in doubt, default to disagreement. False positive on disagreement (extra work) is recoverable; false positive on agreement (rubber-stamping a real bug) is not. Do not introduce a third LLM as a semantic-equivalence judge — that just adds another LLM-as-judge with the same biases.

Step 5 — Report

Hard cap: 2 spawns total — this is a budget ceiling, not an accuracy optimum. The literature on multi-agent verification would support more: BoN-MAV optima are 6-14 verifiers depending on domain (https://arxiv.org/abs/2502.20379); multi-agent debate plateaus at 5-10 agents (https://arxiv.org/abs/2305.14325); self-consistency knee is at 10-20 same-model samples (https://arxiv.org/abs/2203.11171). The 2-cap is a cost-of-being-wrong vs cost-of-audit tradeoff for everyday use; for genuinely high-stakes claims where audit cost is dominated by error cost, raise the cap and add more diverse-method spawns. The cap exists because in most practical cases the marginal third spawn doesn't justify its cost — not because more spawns wouldn't help accuracy.

Present the final answer to the user, followed by a single structured audit line:

[audit; falsification=<X>; re-solve=<agree|disagree>; probe=<agree|disagree-counterexample|N/A>; resolution=<...>; warnings=<perplexity|bypass-case|falsification-unclear|multi-turn|none>; test-coverage=<empirical surface, if partial>]

Mandatory rules — do not violate

1. Verbatim problem transcription. Paraphrasing destroys the audit. Multi-turn → see degraded mode below.

2. Never show the draft to a re-solve or probe subagent. Core anchoring defense.

3. Never use adversarial framing on the subagent. Just "solve" or "trace" or "construct" — not "audit / critique / find errors / verify."

4. Report disagreement honestly. Disagreement must surface in the audit line, even when the draft turns out right.

5. Cross-family setup: calibrated recommendation. The empirical effect is more bounded than commonly claimed:

   • Cross-family nearly eliminates judge bias (self-preference, preference-leakage): drops from 28-37% same-model to ~±1.5% truly cross-family (Preference Leakage, ICLR 2026, https://arxiv.org/abs/2502.01534).
   • Cross-family roughly halves error correlation: same-family pairwise ρ ~0.7-0.8, cross-family ~0.4-0.5 (https://arxiv.org/abs/2602.08003).
   • Cross-family does NOT eliminate correlated errors. Capability drives correlation more than vendor: two strong models from different vendors can agree on errors at 0.99+ (Correlated Errors, ICML 2025, https://arxiv.org/abs/2506.07962). On hard items, models from different families still fail together.
   • Adversarial inputs that fool one family transfer ~47% to others (https://arxiv.org/abs/2307.15043) — shared learned features are real.

   Use cross-family for bias defense; pair with cross-method (Step 3 probe) for error defense. In Claude Code, configure a custom subagent in .claude/agents/auditor.md pointing at a different model family via MCP. The warnings=perplexity flag fires whenever no cross-family check was performed.

Multi-turn handling

After several turns, no single user message states the full problem; the "original problem" exists only as the main agent's reconstruction. Verbatim transcription is impossible — but blindly paraphrasing destroys the audit. Distinguish three buckets when assembling the subagent prompt:

Bucket	Pass to subagent?	Why
Original user text — every relevant user message	Yes, verbatim	Ground truth for what was asked
Confirmed objective constraints — facts the user explicitly stated or confirmed during the conversation ("yes, PostgreSQL 14"; "input is always positive"; "we don't have GPUs")	Yes, labeled as constraints	Necessary context; lossy to omit
Solving thoughts / draft / hypotheses / framings the main agent generated	No	This is the anchor we are defending against

Subagent prompt structure:

<USER MESSAGES>
{verbatim user messages, in order}
</USER MESSAGES>

<CONFIRMED CONSTRAINTS>
- {constraint user explicitly confirmed}
- ...
</CONFIRMED CONSTRAINTS>

In one short paragraph, state the problem in your own words. Then solve it, returning your answer in this format: <format>.

If the subagent's restated problem differs materially from the main agent's understanding, treat that itself as a finding — re-state the problem to the user for confirmation before continuing.

Note warnings=multi-turn in the audit line. Audit value is reduced; do not claim full confidence on a multi-turn-derived problem statement.

Fallback: design / non-recomputable problems

When the problem has no objective single answer (architectural choice, tradeoff space, taste judgment), neither re-solve nor probe applies. Use two-angle failure-mode enumeration:

• Spawn 1 (technical angle):

  Problem: {verbatim problem statement}
  
  WITHOUT seeing any proposed solution, enumerate the failure modes you would
  worry about for any approach to this problem. For each: name, mechanism,
  concrete symptom. Do not propose a design.
  

• Spawn 2 (different angle — adversarial users, degraded conditions, scale, ops):

  Same problem. Same instructions, but focus on failure modes that arise from
  adversarial input, degraded operating conditions, scale extremes, and
  operational handoffs — modes a purely technical review would miss.
  

Main agent checks own proposal against both enumerated lists.

This is the protocol's most degraded mode. Same-family agents enumerate the same failure modes the main agent would have generated; missing modes — the actually-dangerous ones — share the same blind spot. Mitigation is partial. Consider supplementing with the user's own thinking or external literature; do not present design-fallback as if it were the full audit.

Explicit limits — write these honestly in any user-facing summary

• This protocol reduces error rate; it does not eliminate it.

• There is a mathematical floor on correlated-verifier accuracy. For pairwise correlation ρ > 0, ensemble error converges to a positive constant Φ(Φ⁻¹(1−α)/√ρ) — adding more verifiers cannot drive error to zero (https://arxiv.org/abs/2602.08003). Same-family ρ ~0.7-0.8; cross-family ρ ~0.4-0.5; capability-driven errors are correlated regardless of vendor. This protocol does NOT verify correctness; it reduces error rate by a measurable but bounded amount.

• Independence is multi-level, with empirical bounds. Prompt-level independence (different context) helps. Cross-family weights help mostly with judge bias (preference leakage drops near zero), less with errors on hard items (capability-driven correlation persists). Cross-method probing helps with both. No combination achieves true independence.

• For fully empirically verifiable claims, this protocol is inferior to running the test. The triage gate exists so you don't substitute theory for measurement.

• Multi-turn caveat: verbatim transcription degrades. See Multi-turn handling.

• Design type is the most degraded mode. See note in Design fallback.

• Audit-loop has cost. Each spawn is tokens + latency + subagent capacity. Apply when cost-of-being-wrong > cost-of-audit. For ~$1 mistakes, don't spend $10 on audit. The trigger criteria define a lower bound, not a recommendation to apply mechanically; for trivially low-stakes high-confidence claims even within the criteria, skip.

• Bypass cases where the protocol is degraded value:

  • Frontier-novel claims (no training-data ground truth for either agent).
  • Self-consistent fabrications (both agents fabricate consistently from same priors).
  • Aesthetic / taste judgments not amenable to failure-mode enumeration.
  • Falsification shape uncertain after honest characterization.

  When any of these apply, add the relevant warning to the audit line and note the specific case in the user-facing summary. Do not silently report [audit ...] as if a real verification happened.

Handshake with red-team-process memory

• red-team-process: quantitative / external-fact claims → measurement-first + PROVENANCE BLOCK.
• audit-loop (this skill): algorithmic / mechanism / design correctness without full empirical access → independent re-solve + cross-method probe.
• The triage gate routes between them. The two protocols do not overlap; partially-empirical claims trigger both (tests for what's testable, audit-loop for the rest).