Audit loop

Audit loop

A draft answer is verified by an independent re-solve plus a cross-method probe, not by critique. The spawned subagents never see the draft; they work the original problem from independent angles. The three independent results (draft, re-solve, probe) are then mechanically compared.

Why re-solve, not critique

Empirical research on LLM-as-judge shows critique-of-draft has structural failure modes that cannot be patched by prompt engineering:

• Refinement-aware bias — same content scored higher when framed as "revised."
• CoT trust — judges believe shown reasoning traces as ground truth.
• Sycophancy — multi-turn argument produces 3× more flips.
• Answer wavering — multi-round critique echo-chambers rather than converging.
• Self-preference / perplexity bias — same-family model under-flags errors typical of its own training distribution.

See References below for citations. Recomputation/replication is the load-bearing mechanism in every mature human audit domain (reperformance > inquiry in financial audit; replication > peer review; kernel-check > read-the-proof). Step-checking inherits the auditee's blind spots.

Triage gate

Claim type	Route
Quantitative / external-fact claim	→ use the user's red-team-process (measurement-first + PROVENANCE BLOCK). Stop.
Fully covered by tests, or cheap to test exhaustively	→ run the tests. Skip audit-loop.
Partially testable — some empirical surface but invariants, adversarial inputs, or edge cases lie outside test coverage	→ run available tests AND apply audit-loop to the uncovered surface. Record both in the audit line.
Pure reasoning, no empirical surface	→ full audit-loop.
Lookup, opinion, or trivial	→ skip.

The middle row matters. Most algorithm-correctness claims sit here: you can test typical inputs, but invariants and adversarial behavior aren't covered. Don't let the triage push such claims fully into either column.

Protocol

Step 1 — Characterize (internal scratch, do not display)

• CLAIM: one sentence stating what the answer asserts.
• FALSIFICATION SHAPE: what would count as proof this is wrong? Pick the most concrete form:
  • A specific input producing the wrong output
  • A counterexample to a property
  • An independent recomputation yielding a different number
  • A construction that breaks an invariant
  • An attack that breaches a safety claim
• If the falsification shape is unclear after honest thought, note warnings=falsification-unclear; protocol value is degraded.

Step 2 — Spawn 1: Independent re-solve

Codex only spawns subagents when explicitly asked, so this skill is the explicit request. Tell the user out loud, e.g., "Spawning one agent to independently re-solve, plus one cross-method probe, before I deliver."

If a custom auditor agent is defined (see Optional custom agent), use it; else use the built-in default with model_reasoning_effort = "high". Task prompt structure:

Solve the following problem. Return your answer in this format: <format>.

<PROBLEM>
{user's original problem statement, transcribed verbatim — DO NOT paraphrase}
</PROBLEM>

These are construction rules for YOU (the main agent), not text to put in the subagent's prompt. The subagent should see only the minimal "solve" framing above. Explicit mentions of forbidden entities (the draft, the audit context, the verification framing) trigger priming failure: naming X is the dominant way "don't do X" leaks X into the output — ~87.5% of negation violations come from explicit mention of the forbidden entity (Rana 2026, https://arxiv.org/pdf/2601.08070). The fix is structural omission, not negative instruction: don't include those entities in the prompt at all.

Construction rules:

• Transcribe verbatim. Paraphrasing smuggles in main-agent framing and loses constraints. For multi-turn conversations where no single user message states the full problem, see Multi-turn handling below.
• Do NOT include the draft, your reasoning, or any framing about audit/verification in the prompt.
• Do NOT use "audit / critique / find errors / verify" wording in the prompt. Just "solve."
• Specify the output format so comparison is mechanical (see Equivalence semantics below).

Step 3 — Spawn 2: Cross-method probe

Spawn 2 is always a probe via different method, not a third re-solve. This is empirically well-supported:

• BoN-MAV (https://arxiv.org/abs/2502.20379, Table 4): diverse verifiers consistently outperform repeated queries to the best single verifier.
• Diversity of Thought (https://arxiv.org/abs/2310.07088): +10-30 percentage point gains over same-prompt self-consistency.
• Mechanism: same-family models agree on wrong answers ~60% of the time when both wrong, vs ~33% by chance (Correlated Errors, https://arxiv.org/abs/2506.07962). A same-method re-solve inherits that coupling.

A probe via the FALSIFICATION SHAPE breaks the coupling. (Caveat: whether the probe is most valuable on agreement or on disagreement is a design hypothesis — the literature supports method diversity, but the specific allocation has no head-to-head study.)

Falsification shape	Probe task
Specific input → wrong output	"Trace this algorithm/computation on inputs [concrete edge cases]; return outputs."
Counterexample to a property	"Find an input that violates property Y, or after honest effort report 'no case found.'"
Independent recomputation	"Compute X using [different decomposition / different base rate / Fermi-style buildup]; return result."
Construction breaking invariant	"Construct an input that would break invariant Z."
Attack breaching safety claim	"Construct an attack that violates safety property W, or report 'no attack found' after honest effort."
No clean probe (mostly design problems)	See Design fallback below.

Probe subagent gets concrete inputs but never the draft.

Step 4 — Three-way comparison

Compare draft, Spawn 1 (re-solve), and Spawn 2 (probe).

Pattern	Action
All three agree	Strong evidence. Ship.
Draft + re-solve agree; probe finds concrete counterexample	Probe is independent constructive evidence. Revise per probe.
Draft + probe agree; re-solve disagrees	Re-solve was an outlier; ship draft.
Re-solve + probe agree; draft disagrees	Strong signal draft is wrong. Revise to re-solve / per probe.
All three disagree	Admit uncertainty. Report all three to user.

This is "third independent angle, not vote" — the probe is constructive evidence (counterexample / trace / alternative computation), not an averaged opinion.

Equivalence semantics

String-level comparison is too brittle (0.5 vs 1/2 vs ½; lists in different order; algorithms phrased differently but with identical I/O). The main agent applies documented equivalence normalizations before declaring agreement:

• Numbers: canonical decimal at a stated precision; equivalent fractions; equivalent units.
• Sets / unordered lists: order-insensitive.
• Ordered sequences: order-sensitive (state which it is in the requested output format).
• Algorithms / code: identical I/O behavior on the probe inputs counts as equivalent, regardless of phrasing.
• Natural-language claims: equivalence requires exact semantic match on the load-bearing parts; if it requires interpretation, that's disagreement.

When in doubt, default to disagreement. False positive on disagreement (extra work) is recoverable; false positive on agreement (rubber-stamping a real bug) is not. Do not introduce a third LLM as a semantic-equivalence judge — that just adds another LLM-as-judge with the same biases.

Step 5 — Report

Hard cap: 2 spawns total — this is a budget ceiling, not an accuracy optimum. The literature on multi-agent verification would support more: BoN-MAV optima 6-14 (https://arxiv.org/abs/2502.20379); debate plateau 5-10 (https://arxiv.org/abs/2305.14325); self-consistency knee 10-20 (https://arxiv.org/abs/2203.11171). The 2-cap is a cost-of-being-wrong vs cost-of-audit tradeoff for everyday use; for genuinely high-stakes claims where audit cost is dominated by error cost, raise the cap and add diverse-method spawns. The cap exists because in most practical cases the third spawn doesn't justify its cost — not because more spawns wouldn't help.

Present the final answer to the user, followed by a single structured audit line:

[audit; falsification=<X>; re-solve=<agree|disagree>; probe=<agree|disagree-counterexample|N/A>; resolution=<...>; warnings=<perplexity|bypass-case|falsification-unclear|multi-turn|none>; test-coverage=<empirical surface, if partial>]

Mandatory rules — do not violate

1. Verbatim problem transcription. Multi-turn → see degraded mode below.

2. Never show the draft to a re-solve or probe subagent. Core anchoring defense.

3. Never use adversarial framing on the subagent. Just "solve" or "trace" or "construct" — not "audit / critique / find errors / verify."

4. Report disagreement honestly. Disagreement must surface in the audit line, even when the draft turns out right.

5. Cross-family setup: calibrated recommendation. The empirical effect is more bounded than commonly claimed:

   • Cross-family nearly eliminates judge bias (self-preference, preference-leakage): drops from 28-37% same-model to ~±1.5% truly cross-family (Preference Leakage, ICLR 2026, https://arxiv.org/abs/2502.01534).
   • Cross-family roughly halves error correlation: same-family ρ ~0.7-0.8, cross-family ~0.4-0.5 (https://arxiv.org/abs/2602.08003).
   • Cross-family does NOT eliminate correlated errors. Capability drives correlation more than vendor: two strong models from different vendors can agree on errors at 0.99+ (Correlated Errors, ICML 2025, https://arxiv.org/abs/2506.07962).
   • Adversarial inputs transfer ~47% across families (https://arxiv.org/abs/2307.15043).

   Use cross-family for bias defense; pair with cross-method (Step 3 probe) for error defense. Codex makes cross-family easy via the model field on ~/.codex/agents/auditor.toml. The warnings=perplexity flag fires whenever no cross-family check was performed.

Multi-turn handling

After several turns, no single user message states the full problem; the "original problem" exists only as the main agent's reconstruction. Verbatim transcription is impossible — but blindly paraphrasing destroys the audit. Distinguish three buckets:

Bucket	Pass to subagent?	Why
Original user text — every relevant user message	Yes, verbatim	Ground truth for what was asked
Confirmed objective constraints — facts the user explicitly stated or confirmed during the conversation	Yes, labeled as constraints	Necessary context; lossy to omit
Solving thoughts / draft / hypotheses / framings the main agent generated	No	This is the anchor we are defending against

Subagent prompt structure:

<USER MESSAGES>
{verbatim user messages, in order}
</USER MESSAGES>

<CONFIRMED CONSTRAINTS>
- {constraint user explicitly confirmed}
- ...
</CONFIRMED CONSTRAINTS>

In one short paragraph, state the problem in your own words. Then solve it, returning your answer in this format: <format>.

If the subagent's restated problem differs materially from the main agent's understanding, treat that itself as a finding — re-state to the user for confirmation. Note warnings=multi-turn in the audit line.

Fallback: design / non-recomputable problems

When the problem has no objective single answer, neither re-solve nor probe applies. Use two-angle failure-mode enumeration:

• Spawn 1 (technical angle):

  Problem: {verbatim problem statement}
  
  WITHOUT seeing any proposed solution, enumerate the failure modes you would
  worry about for any approach to this problem. For each: name, mechanism,
  concrete symptom. Do not propose a design.
  

• Spawn 2 (different angle — adversarial users, degraded conditions, scale, ops):

  Same problem. Same instructions, but focus on failure modes that arise from
  adversarial input, degraded operating conditions, scale extremes, and
  operational handoffs — modes a purely technical review would miss.
  

Main agent checks own proposal against both lists.

This is the protocol's most degraded mode. Same-family agents enumerate the same failure modes the main agent would have generated; missing modes — the actually-dangerous ones — share the same blind spot. Mitigation is partial. Supplement with the user's own thinking or external literature; do not present design-fallback as if it were full audit.

Optional custom agent

Define a dedicated re-solver at ~/.codex/agents/auditor.toml. Instructions are NOT adversarial — adversarial framing produces hallucinated critique:

name = "auditor"
description = "Independent re-solver for the audit-loop skill. Solves a problem from scratch for cross-checking, with no view of any prior draft."
model = "gpt-5.5"  # pin a different model family here if available — cross-family beats cross-prompt
model_reasoning_effort = "high"
sandbox_mode = "read-only"
developer_instructions = """
You will be given a problem statement. Solve it from scratch in the requested
output format. Do not assume any prior draft exists. Do not critique anything.
If the problem is ambiguous, state the ambiguity in your answer rather than
guessing. If you would naturally choose between several solution methods,
prefer the one less likely to be the obvious default — diversity of approach
is the reason you were spawned.
"""
nickname_candidates = ["Cassandra", "Sokrates", "Vega"]

Explicit limits — write these honestly in any user-facing summary

• This protocol reduces error rate; it does not eliminate it.

• There is a mathematical floor on correlated-verifier accuracy. For pairwise correlation ρ > 0, ensemble error converges to a positive constant Φ(Φ⁻¹(1−α)/√ρ) — adding more verifiers cannot drive error to zero (https://arxiv.org/abs/2602.08003). Same-family ρ ~0.7-0.8; cross-family ρ ~0.4-0.5; capability-driven errors are correlated regardless of vendor. This protocol does NOT verify correctness; it reduces error rate by a measurable but bounded amount.

• Independence is multi-level, with empirical bounds. Prompt-level (different context) helps. Cross-family weights help mostly with judge bias (preference leakage drops near zero), less with errors on hard items (capability-driven correlation persists). Cross-method probing helps with both. No combination achieves true independence. Codex's model field on auditor.toml lets you pin a different model — use it for high-stakes work.

• For fully empirically verifiable claims, this protocol is inferior to running the test. The triage gate exists so you don't substitute theory for measurement.

• Multi-turn caveat: verbatim transcription degrades. See Multi-turn handling.

• Design type is the most degraded mode. See note above.

• Audit-loop has cost. Each spawn is tokens + latency + subagent capacity. Apply when cost-of-being-wrong > cost-of-audit. The trigger criteria define a lower bound, not a recommendation to apply mechanically; for trivially low-stakes high-confidence claims, skip.

• Bypass cases:

  • Frontier-novel claims (no training-data ground truth).
  • Self-consistent fabrications (both agents fabricate consistently).
  • Aesthetic / taste judgments.
  • Falsification shape uncertain after honest characterization.

  When any apply, add the relevant warning and note in the user-facing summary.

Handshake with red-team-process

• red-team-process: quantitative / external-fact claims → measurement-first + PROVENANCE BLOCK.
• audit-loop (this skill): algorithmic / mechanism / design correctness without full empirical access → independent re-solve + cross-method probe.
• The triage gate routes between them. The two protocols do not overlap; partially-empirical claims trigger both.