Audit loop

Audit loop

A draft answer is verified by an independent re-solve plus a cross-method probe, not by critique. The spawned subagents never see the draft; they work the original problem from independent angles. The three independent results (draft, re-solve, probe) are then mechanically compared.

Why re-solve, not critique

Empirical research on LLM-as-judge shows critique-of-draft has structural failure modes that cannot be patched by prompt engineering:

• Refinement-aware bias — same content scored higher when framed as "revised."
• CoT trust — judges believe shown reasoning traces as ground truth.
• Sycophancy — multi-turn argument produces 3× more flips.
• Answer wavering — multi-round critique echo-chambers rather than converging.
• Self-preference / perplexity bias — same-family model under-flags errors typical of its own training distribution.

See References below for citations. Recomputation/replication is the load-bearing mechanism in every mature human audit domain (reperformance > inquiry in financial audit; replication > peer review; kernel-check > read-the-proof). Step-checking inherits the auditee's blind spots.

Triage gate

Claim type	Route
Quantitative / external-fact claim	→ use the user's red-team-process (measurement-first + PROVENANCE BLOCK). Stop.
Fully covered by tests, or cheap to test exhaustively	→ run the tests. Skip audit-loop.
Partially testable — some empirical surface but invariants, adversarial inputs, or edge cases lie outside test coverage	→ run available tests AND apply audit-loop to the uncovered surface. Record both in the audit line.
Pure reasoning, no empirical surface	→ full audit-loop.
Lookup, opinion, or trivial	→ skip.

The middle row matters. Most algorithm-correctness claims sit here.

Protocol

Step 1 — Characterize (internal scratch, do not display)

• CLAIM: one sentence stating what the answer asserts.
• FALSIFICATION SHAPE: what would count as proof this is wrong? Most concrete form: specific input → wrong output; counterexample to property; independent recomputation differs; construction breaks invariant; attack breaches safety claim.
• If falsification shape is unclear, note warnings=falsification-unclear; protocol value is degraded.

Step 2 — Spawn 1: Independent re-solve

Call sessions_spawn with context: "isolated" (default — never fork, which would leak the draft). Then call sessions_yield to wait for completion — do NOT poll subagents list or sessions_history.

sessions_spawn({
  task: <re-solve prompt, problem verbatim — see below>,
  taskName: "audit_resolve",
  label: "Independent re-solve",
  context: "isolated",
  runTimeoutSeconds: 600
})
sessions_yield()

Task prompt (what the subagent actually sees):

Solve the following problem. Return your answer in this format: <format>.

<PROBLEM>
{user's original problem statement, transcribed verbatim — DO NOT paraphrase}
</PROBLEM>

The construction rules below are for YOU (the main agent), NOT text to put in the task field. The subagent should see only the minimal "solve" framing above. Explicit mentions of forbidden entities (the draft, the audit context, the verification framing) trigger priming failure: naming X is the dominant way "don't do X" leaks X into the output — ~87.5% of negation violations come from explicit mention of the forbidden entity (Rana 2026, https://arxiv.org/pdf/2601.08070). The fix is structural omission: don't include those entities in the task at all.

Construction rules:

• Transcribe verbatim. For multi-turn conversations, see Multi-turn handling.
• Do NOT include the draft, your reasoning, or any audit/verification framing in the task.
• Do NOT use "audit / critique / find errors / verify" wording in the task. Just "solve."
• Specify the output format so comparison is mechanical (see Equivalence semantics).

Step 3 — Spawn 2: Cross-method probe

Spawn 2 is always a probe via different method, not a third re-solve. Empirically well-supported:

• BoN-MAV (https://arxiv.org/abs/2502.20379, Table 4): diverse verifiers consistently beat repeated queries to the best single verifier.
• Diversity of Thought (https://arxiv.org/abs/2310.07088): +10-30 pp gains over same-prompt self-consistency.
• Mechanism: same-family models agree on wrong answers ~60% of the time when both wrong, vs ~33% by chance (Correlated Errors, https://arxiv.org/abs/2506.07962).

A probe via the FALSIFICATION SHAPE breaks the coupling. (Caveat: whether the probe is most valuable on agreement or disagreement is a design hypothesis — method diversity is supported, the specific allocation has no head-to-head study.)

Falsification shape	Probe task
Specific input → wrong output	"Trace this algorithm/computation on inputs [edge cases]; return outputs."
Counterexample to a property	"Find an input that violates property Y, or report 'no case found' after honest effort."
Independent recomputation	"Compute X using [different decomposition / different base rate / Fermi-style]; return result."
Construction breaking invariant	"Construct an input that would break invariant Z."
Attack breaching safety claim	"Construct an attack that violates safety property W, or report 'no attack found'."
No clean probe (mostly design)	See Design fallback.

Spawn:

sessions_spawn({
  task: <probe task with specific inputs>,
  taskName: "audit_probe",
  label: "Cross-method probe",
  context: "isolated",
  runTimeoutSeconds: 600
})
sessions_yield()

Probe subagent gets concrete inputs but never the draft.

Step 4 — Three-way comparison

Compare draft, re-solve, probe.

Pattern	Action
All three agree	Strong evidence. Ship.
Draft + re-solve agree; probe finds counterexample	Probe is independent constructive evidence. Revise per probe.
Draft + probe agree; re-solve disagrees	Re-solve was outlier; ship draft.
Re-solve + probe agree; draft disagrees	Strong signal draft is wrong. Revise.
All three disagree	Admit uncertainty. Report all three to user.

"Third independent angle, not vote" — probe is constructive evidence, not averaged opinion.

Equivalence semantics

String-level comparison is too brittle (0.5 vs 1/2; lists in different order; algorithms phrased differently). The main agent applies documented equivalence normalizations:

• Numbers: canonical decimal at stated precision; equivalent fractions; equivalent units.
• Sets / unordered lists: order-insensitive.
• Ordered sequences: order-sensitive (state in the format spec).
• Algorithms / code: identical I/O on probe inputs counts as equivalent.
• Natural-language claims: exact semantic match on load-bearing parts; if interpretation needed, that's disagreement.

When in doubt, default to disagreement. False-positive disagreement is recoverable; false-positive agreement is not. Do not introduce a third LLM as semantic-equivalence judge — same biases.

Step 5 — Report

Hard cap: 2 spawns total — this is a budget ceiling, not an accuracy optimum. The literature on multi-agent verification would support more: BoN-MAV optima 6-14 (https://arxiv.org/abs/2502.20379); debate plateau 5-10 (https://arxiv.org/abs/2305.14325); self-consistency knee 10-20 (https://arxiv.org/abs/2203.11171). The 2-cap is a cost-of-being-wrong vs cost-of-audit tradeoff for everyday use; for genuinely high-stakes claims, raise the cap and add diverse-method spawns. The cap exists because in most practical cases the third spawn doesn't justify its cost — not because more spawns wouldn't help.

Reply to the user via your normal message tool with the final answer + structured audit line:

[audit; falsification=<X>; re-solve=<agree|disagree>; probe=<agree|disagree-counterexample|N/A>; resolution=<...>; warnings=<perplexity|bypass-case|falsification-unclear|multi-turn|none>; test-coverage=<if partial>]

Mandatory rules — do not violate

1. Verbatim problem transcription. Multi-turn → see degraded mode below.

2. Never show the draft to a re-solve or probe subagent. Core anchoring defense.

3. Never use adversarial framing. Just "solve" or "trace" or "construct."

4. Report disagreement honestly.

5. Cross-family setup: calibrated recommendation. The empirical effect is more bounded than commonly claimed:

   • Cross-family nearly eliminates judge bias (self-preference, preference-leakage): drops from 28-37% same-model to ~±1.5% truly cross-family (Preference Leakage, ICLR 2026, https://arxiv.org/abs/2502.01534).
   • Cross-family roughly halves error correlation: same-family ρ ~0.7-0.8, cross-family ~0.4-0.5 (https://arxiv.org/abs/2602.08003).
   • Cross-family does NOT eliminate correlated errors. Capability drives correlation more than vendor: two strong models from different vendors can agree on errors at 0.99+ (Correlated Errors, ICML 2025, https://arxiv.org/abs/2506.07962).
   • Adversarial inputs transfer ~47% across families (https://arxiv.org/abs/2307.15043).

   Use cross-family for bias defense; pair with cross-method (Step 3 probe) for error defense. OpenClaw's agents.defaults.subagents.model (or per-agent agents.list[].subagents.model) can point to any configured provider — this is the best of the three platforms for cross-family setup. The warnings=perplexity flag fires whenever no cross-family check was performed.

6. Do not poll for completion. Use sessions_yield after spawning. Polling subagents list, sessions_history, or shell sleeps is explicitly forbidden by OpenClaw's subagent guidance.

7. Do not use context: "fork" — it leaks the requester's transcript (including the draft) into the subagent. isolated is required.

Multi-turn handling

After several turns, no single user message states the full problem; the "original problem" exists only as the main agent's reconstruction. Verbatim transcription is impossible — but blindly paraphrasing destroys the audit. Distinguish three buckets in the task field:

Bucket	Pass to subagent?	Why
Original user text — every relevant user message	Yes, verbatim	Ground truth for what was asked
Confirmed objective constraints — facts the user explicitly stated or confirmed	Yes, labeled as constraints	Necessary context
Solving thoughts / draft / hypotheses / framings main agent generated	No	This is the anchor we are defending against

Task structure:

<USER MESSAGES>
{verbatim user messages}
</USER MESSAGES>

<CONFIRMED CONSTRAINTS>
- {explicit user-confirmed constraint}
- ...
</CONFIRMED CONSTRAINTS>

In one short paragraph, state the problem in your own words. Then solve it, returning your answer in this format: <format>.

If subagent's restated problem differs materially from main agent's understanding, treat that itself as a finding — re-state to user for confirmation. Note warnings=multi-turn.

Fallback: design / non-recomputable problems

When problem has no objective single answer, use two-angle failure-mode enumeration:

Spawn 1 (technical):

sessions_spawn({
  task: "Problem: {verbatim}\n\nWITHOUT seeing any proposed solution, enumerate the failure modes you would worry about for any approach to this problem. For each: name, mechanism, concrete symptom. Do not propose a design.",
  taskName: "audit_enum_tech",
  label: "Failure modes — technical angle",
  context: "isolated"
})
sessions_yield()

Spawn 2 (different angle):

sessions_spawn({
  task: "Same problem. Same instructions, but focus on failure modes from adversarial input, degraded operating conditions, scale extremes, and operational handoffs — modes a purely technical review would miss.",
  taskName: "audit_enum_ops",
  label: "Failure modes — ops/adversarial angle",
  context: "isolated"
})
sessions_yield()

Main agent checks own proposal against both lists.

This is the protocol's most degraded mode. Same-family agents enumerate the same failure modes the main agent would have generated; missing modes — the actually-dangerous ones — share the same blind spot. Supplement with user's own thinking or external literature; do not present design-fallback as full audit.

Explicit limits — write these honestly in user-facing replies

• This protocol reduces error rate; it does not eliminate it.
• Independence is structural-prompt-level, not weight-level. Same model family → shared training data → correlated blind spots. OpenClaw's per-agent subagents.model lets you pin a different model family — use it for high-stakes work.
• For fully empirically verifiable claims, this protocol is inferior to running the test.
• Multi-turn caveat: verbatim transcription degrades.
• Design type is the most degraded mode.
• Audit-loop has cost. Each spawn is tokens + latency + subagent capacity. Apply when cost-of-being-wrong > cost-of-audit. The trigger criteria define a lower bound, not a recommendation to apply mechanically.
• Bypass cases: frontier-novel claims; self-consistent fabrications; aesthetic judgments; falsification shape uncertain. Add relevant warning to audit line and note in user-facing reply.

Cost note

Two spawns at runTimeoutSeconds: 600 is the budget ceiling. For high volume, pin a cheaper auditor model via agents.defaults.subagents.model — main agent stays on higher-quality model. Pinning a different family on the auditor lane is the highest-leverage configuration change.

Handshake with red-team-process

• red-team-process: quantitative / external-fact claims → measurement-first + PROVENANCE BLOCK.
• audit-loop (this skill): algorithmic / mechanism / design correctness without full empirical access → independent re-solve + cross-method probe.
• Triage gate routes between them. Partially-empirical claims trigger both.