Second opinion bench

Second Opinion Bench

Use this skill when asked to produce benchmark verdicts with the current model or to grade verdicts already produced elsewhere.

Integrity rules

1. The harness is model-agnostic and local. Never add a live model call to src/; verdict generation happens in the host agent.
2. During review, read only fixtures/diffs/*/change.diff. Do not read any ground-truth.json, prior verdict, result, or example report.
3. Review every fixture independently. Do not infer that each diff must contain exactly one issue, even though the current corpus does.
4. Freeze the raw finding summaries before opening ground truth. After freezing, map only an already-recorded finding to a planted defect ID when the semantics clearly match. Never create, expand, or delete a finding because of ground truth.
5. Give every unmatched finding a stable reviewer-proposed ID namespaced to the fixture. The grader must be allowed to count it as a false positive.
6. Never describe a simulated, hand-authored, incomplete, or ground-truth-exposed run as measured model performance.

Produce verdicts with this model

Choose a reviewer label that identifies the actual model/version and material review configuration. Create reviews/<reviewer>/.

For each fixture directory, in isolation:

1. Read change.diff only.
2. Look for correctness, edge-case, security, monetary, concurrency, and behavior-contract defects. Record only defects supported by the diff.
3. Freeze a list of concise summaries. An empty list is allowed.
4. Only now read that fixture’s ground-truth.json to map a frozen summary to an exact planted ID. Preserve unmatched summaries under reviewer-proposed IDs.
5. Save reviews/<reviewer>/<fixtureId>.json conforming to schemas/verdict.schema.json:

{
  "reviewer": "provider-model-version-prompt-v1",
  "fixtureId": "edge-empty-average",
  "foundDefects": [
    {
      "id": "edge-empty-average/reduce-without-seed",
      "summary": "Empty input calls reduce without an initial value."
    }
  ]
}

Write one verdict for every fixture, including "foundDefects": [] when no defect was found. Do not copy the demonstration verdicts.

To test a different model, run this protocol again in a fresh agent context using that model. Do not expose the new reviewer to another reviewer’s files or results until its verdicts are complete.

Grade sealed verdicts

After all reviewers have sealed their files, run:

node src/run-bench.mjs \
  --reviews reviews \
  --output bench-result.json \
  --report bench-report.md

The command reads JSON files only; it does not contact a model or provider. Treat any coverage warning as an incomplete run, not as a zero-finding review.

Report honestly

State:

• reviewer labels and enough model/configuration detail to reproduce them;
• fixture coverage for every reviewer;
• overall and per-class recall with numerators and denominators;
• false-positive counts;
• pair union recall;
• both directional uplifts: what B adds after A and what A adds after B;
• whether any verdict was simulated, hand-authored, post-edited, or exposed to ground truth before findings were frozen;
• the limitations that fixtures are synthetic and the grader matches IDs, not semantics.

Do not say “model B is better” from this corpus alone. A defensible statement is narrower: “On this 12-defect synthetic corpus, under the recorded setup, B added N planted defects that A missed.”