Review cross

Cross-vendor review

Overview

A reviewer from a different vendor, reading the spec and the change, in third person, answering one question: does the implementation violate a numbered rule, or does a test claim to pass without checking?

The verdict gates. It runs twice on identical input, and disagreement is inconclusive, not a coin flip you get to resolve in the direction you prefer.

Why it exists, with the number

Measured 2026-08-04. N=1, no pre-registration, so read it as an observation and not as a result.

One call, roughly 15k tokens, against a session.create spec in a reference service. It found two real things that 26 injected mutants, 15 scenarios and 6 machine-enforced boundary contracts had all missed:

1. The spec rule read "two sessions of the same community cannot share a start time". The table had no community column and the unique index was global, so one community would block another. It lived in the state module, the most protected file in that repository.
2. Two [screen] assertions in the spec with zero tests verifying them.

The reason is structural, not luck. Mutation testing starts from the code, so it inherits whatever the code already got wrong. A reviewer that starts from the spec does not. That is the whole argument for this gate, and it is the reason it cannot be replaced by more mutants.

On a later run against the same behavior it returned one of the two. Take that as the honest shape of the tool: it is a second pair of eyes with a different prior, not a complete checker. The assertion-without-verifier class is covered mechanically by spec-coverage, and cheaply. Run both.

When to use

• Before merging any behavior, as the last gate
• When the same model wrote both the code and its tests
• When a spec rule is stated in prose and you cannot point at the assertion for it

When not to use: as a linter, or for style. It is instructed to ignore naming, structure and taste, and it will.

The rule that makes it a gate and not a vibe

Two runs, byte-identical input.

Outcome	Meaning
Both agree	The verdict stands, and it gates. Exit 0 approved, exit 1 needs changes
Runs disagree	inconclusive, exit 2. Does not consume an attempt. Opens quarantine with an owner and a deadline

A judge that flips on identical input has not judged anything. Picking whichever of the two answers you like, after seeing which is which, is the exact move this whole approach exists to prevent.

The parser fails closed. Unreadable output, or a verdict outside the two allowed values, becomes needs_changes with a meta finding quoting the raw tail. An answer you cannot read is never approval.

How to run

python3 ~/.claude/skills/review-cross/scripts/review-cross.py \
  --spec docs/behaviors/<id>/spec.md \
  --paths <impl.py> <state.py> <test.py> \
  --repo . --runs 2

Or against a diff, which is what you want inside a loop:

git diff <base>...HEAD | python3 .../review-cross.py --spec <spec> --diff - --repo .

--json gives machine-readable output for a harness. --selftest exercises the parser and the gate with no model call and no cost, including the case where the reviewer returns garbage.

Reviewer command is codex by default, overridable with REVIEW_CROSS_CMD. The call is sandboxed read-only and runs with --ignore-user-config, because a reviewer that can write is an author, and the operator's personal config must not leak into the review session.

Quick reference

Finding kind	What it means
spec_violated	A numbered rule says one thing, the implementation does another
assertion_unverified	A test asserts a status or a return and never the effect the spec describes
unspecified_behavior	Real behavior nobody wrote down that breaks in production

Every finding carries path:line. A finding you cannot point at is not a finding.

Common mistakes

Using the same vendor. The whole point is a different prior. If codex is not installed, this gate does not degrade gracefully into the same model reviewing itself, and the script refuses rather than pretending.

Telling the reviewer the code is yours. Third person is not decoration. Attribution changes what the model produces, because in the training distribution a code review is almost always someone reviewing someone else, and almost never a person listing their own mistakes. This claim came from a talk on AI development workflow, and it has not been isolated as a variable by anyone, here or elsewhere. Treat it as untested, and keep it because it is free.

Resolving an inconclusive by re-running until you like it. That is fishing. Quarantine, owner, deadline.

Treating it as complete. It is one reviewer with one prior. Run spec-coverage for the mechanical classes and audit-tests for test quality.

Excuse versus reality

Excuse	Reality
"Mutation score is clean"	Mutation starts from the code and inherits its errors
"The tests pass"	The tests were written by the model that wrote the code
"It only found one thing"	One real defect in the most protected file in the repo
"Two runs is expensive"	Two calls at ~15k tokens against a defect that shipped
"The runs disagreed, I will take the good one"	Then you are the judge, and you skipped the reading

Red flags, stop

• You are about to merge on inconclusive
• You swapped the reviewer for the same vendor because the other was not installed
• You told the reviewer the code was yours
• The gate failed and you are looking for a flag to make it pass