Audit test

Owns: whether an existing passing test is real — confirmed by running one targeted mutation. The verifier. Not this: missing paths or new tests to add → /coverage-review; redundant or stale tests to remove → /prune-tests; whether the source is testable at all → /qa-review.

A green test is not proof. This skill asks the sharpest question about the tests you already have: would this passing test fail if the code it covers broke? If it wouldn't, it's false confidence — it looks like protection but guards nothing.

The trap: an AI can reason a test is fine and be exactly as wrong as the test it's judging. So don't stop at reasoning. For a suspect test, reason out the single most-likely-breaking change to the code, apply that mutation, run just that one test, and report what actually happened — never the whole suite. The strongest honest claim is factual: "I broke the code like this and the test still passed." Whether the mutation was behaviorally meaningful stays a visible human judgment — this skill is a challenger, not an oracle. (See ADR-0001.)

There's a second, subtler failure the mutation alone can't see: a test whose assertion is live but pinned to the wrong value — edited to bless a regression (the fingerprint an AI self-healer leaves when it "fixes" a red test by rewriting the expected value). It kills mutations, so it reads 🟢 — yet it enforces the broken behavior and would reject the real fix. Catching it needs an intent signal, not just a mutation (see ADR-0017).

Every verdict below is this skill's own account of a mutation it actually ran in your session, not an independently verified measurement — there's no executable behind it checking its work, only the Safety rule below and you watching the tool calls go by.

Steps

1. Resolve the target from $ARGUMENTS:

   • a single named test → deep-audit that one test.
   • a test file → triage every test in it, deep-audit only the flagged ones.
   • a test and its code (both paths given) → deep-audit, code-aware. The first-class mode — full confidence needs the code.
   • a directory, glob, or --changed → resolve to a set of test files, then run Batch mode. Discover test files with the multi-ecosystem patterns below; for --changed, list files changed against the merge-base (git diff --name-only main...HEAD) filtered to those same patterns. This is the mode /qa-pass calls over a branch's changed tests.
   • nothing → default to the whole suite (same patterns) and run Batch mode.

   Batch mode runs diagnostic (default — deep-audit only the flagged suspects) or certification (--certify — deep-audit a floor-sized random sample ∪ the suspects, the opt-in path to an honest ship); Batch mode specifies both.

   Test-file discovery patterns (extensible by convention — not JS-only, per ADR-0014):

   • JS/TS: **/*.{spec,test}.{js,jsx,ts,tsx,mjs,cjs}
   • pytest: **/test_*.py, **/*_test.py
   • Go: **/*_test.go
   • JVM: **/*Test.java, **/*Tests.java, **/*Test.kt

   If discovery matches zero test files, the audit is INCONCLUSIVE — no recognized test files: report it and stop. Never treat "nothing found" as "nothing wrong" — a caller like /qa-pass must read INCONCLUSIVE as "the audit did not run," not as a clean result.

2. Read the test(s) fully. If code paths are given, read those too.

3. Triage (cheap, static). For each test, state its behavior contract in one sentence ("what real behavior would break if this failed?"), then smell-check it. The assertion-quality smells — loose, incidental, overmocked (taxonomy 2–4) — are the same static read /coverage-review defines; apply that read here rather than re-deriving it. The taxonomy below adds the smells specific to this skill's question (focal-unit-never-invoked, order-dependent, implementation-coupled, pseudo-tested). What audit-test contributes beyond the static read is Step 4 — escalating the suspects to a live mutation. Only suspicious tests advance; this funnel keeps runtime to a handful of single-test runs.

4. Deep audit (per flagged test). Reason out the single most plausible code change that should make this test fail, then prove it — honoring the Safety rule:

   • Apply the mutation to the source, run just that one test (framework commands: reference/run-one-test.md), record pass/fail, then revert immediately.
   • Test failed as it should → 🟢 killed the proposed mutation (confirmed-solid against this mutation — not a blanket guarantee the test is fine). But before recording 🟢 for a changed test, run the Baseline-lock check: a live assertion can still be pinned to a regressed value it was edited to match — if it fires, report ⚠️ Baseline-lock suspected, not 🟢.
   • Test still passed → candidate 🔴 — do not record it yet. Run the Reachability check first: only a survival that clears reachability is 🔴 Confirmed false-confidence; a survival that fails reachability is 🟡 — the mutation never reached the running app, so the test target is stale, not the test confirmed hollow.
   • If the code can't be run (no runnable env, missing deps), do not guess a Confirmed verdict — fall back to the mutation thought experiment and label it 🟡 Likely.

5. Classify each finding with the failure taxonomy and write the report. A characterization test (one deliberately pinning current behavior, even quirky behavior, so a refactor can't change it silently) is labeled a safety net, not condemned.

6. If --explain is present in $ARGUMENTS, append a "Why This Matters" section (see Explain Mode). Otherwise omit it — default output stays lean.

7. If --digest is present in $ARGUMENTS, emit the shared digest card instead of the report above — the same verdicts, trimmed to the top three findings as risk / evidence / action / confidence. The verdict maps straight onto the label and never moves: 🔴/🟢 are Confirmed (a mutation ran and its outcome was observed), 🟡 and ⚠️ are Likely, triaged-but-never-mutated is Unexamined and belongs in the header's denominator, not in a card.

Verdicts

Reuses /qa-pass's three states, scoped to a single test, plus one suspicion flag (⚠️) for the failure a mutation can't see:

• 🔴 Confirmed false-confidence — mutation ran, test stayed green, and the reachability check confirmed the harness is source-live. Factual, execution-confirmed.
• 🟡 Likely false-confidence — reasoned only; code wasn't runnable or the mutation didn't reach the running app (stale/remote harness), so this is short of proof.
• 🟢 Killed the proposed mutation — a mutation actually ran and the test failed as it should. Confirmed-solid against that mutation; not a blanket "this test is fine." Reasoning alone never earns 🟢 — an inability to devise a breaking mutation is 🟡 Likely (if reasoned about) or Unexamined (if it never advanced past triage), never confirmed-solid (ADR-0039).
• ⚠️ Baseline-lock suspected — the assertion is live (it kills mutations) but appears pinned to a value the code was changed to produce, not its intended behavior. Neither a clean 🟢 nor a hollow 🔴: a distinct suspicion that needs the reviewer (or an intent source) to confirm the intended value. A caller reads it as caution, never a pass. (See ADR-0017.)

Failure taxonomy ("How it fails")

1. Focal unit never invoked — the function under test is never actually called (setup runs, assertions check something else).
2. Overmocked / interaction-only — asserts a collaborator was called, never that the real behavior happened. Verifies the mock, not the code.
3. Incidental assertion — checks something true regardless of whether the behavior works (a field that's always set, a status that's always 200).
4. Loose assertion — a check that can't meaningfully fail (toBeDefined(), toBeTruthy() on a whole object). (See GLOSSARY.md: Loose Assertion.)
5. Order-dependent assertion — passes only because of test execution order or shared state, not because the behavior is correct.
6. Implementation-coupled — asserts how the code works (internal calls, private shape) rather than what it guarantees; passes even when the guarantee is broken. (Needs code+test mode to judge.)
7. Pseudo-tested — the strongest, execution-confirmed case: the code can be broken arbitrarily and the test never notices.
8. Baseline-locked — a live assertion pinned to the wrong value: it kills mutations but was edited to match a regression, so it enforces the broken behavior and would reject the fix. Distinct from 1–7 (which guard nothing); this one guards the wrong thing. Detected via the Baseline-lock check, not the mutation alone.

Output Format

Single-test / flagged-test entry:

## audit-test: "rejects overlapping bookings"
**Verdict:** 🔴 Confirmed false-confidence
**How it fails:** Overmocked — asserts save() was called, never that the overlap was rejected
**Proof:** commented out the overlap guard (booking.ts:34) → ran this test → still passed
**A real test would:** assert the 2nd booking is rejected with 409, not that save() ran

**Next:** strengthen the assertion above, then re-run `/audit-test tests/booking.spec.ts` — the fix is only real when the mutation dies

For a 🟡 verdict, replace Proof with Reasoning and say why the code couldn't be run (or, if no plausible breaking mutation was found, say so — that's a 🟡, not a 🟢). For a 🟢, state briefly what mutation you ran and how it made the test fail. For ⚠️ Baseline-lock suspected, show the intent signal — the co-changed assertion (old → new expected value) or the in-code source of truth it contradicts — and the remedy:

## audit-test: "renders the initial deck"
**Verdict:** ⚠️ Baseline-lock suspected
**How it fails:** Baseline-locked — assertion loosened in lockstep with the code change it should catch
**Signal:** assertion `toHaveCount(12)` → `toHaveCount(10)`, co-changed with `Main.reducer.ts` slicing the deck to 10; `robots.ts` declares 12 cards / 6 pairs
**Remedy:** restore `toHaveCount(12)`, or — if a 10-card deck is intended — update `robots.ts`/the deck's declared size to match. Confirm the intended count before merging.

Single-test mode always shows its verdict, including 🟢. Batch mode reports a provenance tally instead — Unexamined tests are never folded into the confirmed-solid greens; see reference/batch-mode.md for its format (ADR-0013).

Close every report — single-test, batch, or --digest (shared card) — with the one-line Next: footer shown above. Pick the row for the verdict you actually reported: 🔴/⚠️ route back into a re-run after the fix; 🟡 routes to making the code runnable (or to /audit-orchestrator to pick a prover your stack can run); 🟢 or a clean batch routes on to /qa-pass, then /gate.

Structured emission (--emit-json=<path>) — for the Gate skill

When --emit-json=<path> is present in $ARGUMENTS, also write the run's provenance tally as a small gate-audit-test/v0.4 JSON file at that path — in addition to the human report, which is unchanged. This is the parsed credibility input the Gate skill (/gate) ingests: a parsed confirmed-clean verdict is the only thing that lets Gate lift a release decision to ship (the B→A graduation, ADR-0029). Emit only the counts — never a prose scrape and never a confidence number; Gate derives the category itself (schema):

{
  "schema": "gate-audit-test/v0.4",
  "producer": "audit-test",
  "scope": "--changed",
  "commitSha": "a1b2c3d4e5f6...",
  "dirty": false,
  "audited": 47,
  "deepAudited": 5,
  "confirmedSolid": 2,
  "confirmedHollow": 1,
  "likelyHollow": 1,
  "baselineLock": 1,
  "unexamined": 42,
  "runs": [
    {
      "test": "<file>::<test name>",
      "mutation": "one-line description of the change applied",
      "command": "the exact single-test command run (framework-specific — reference/run-one-test.md)",
      "outcome": "killed",
      "exitCode": 1
    }
  ]
}

(runs[] trimmed to one representative entry above — a real emission carries one record per confirmed-solid/confirmed-hollow test, so confirmedSolid: 2, confirmedHollow: 1 here would carry three.)

The counts map 1:1 to the batch tally's verdict classes: confirmedSolid = 🟢, confirmedHollow = 🔴, likelyHollow = 🟡, baselineLock = ⚠️, unexamined = triaged-but-never-mutated; deepAudited = how many advanced past triage, audited = total triaged. The counts must be truthful and consistent — in particular, if you deep-audited nothing, deepAudited is 0 (Gate derives unexamined → the run can never reach ship). For an INCONCLUSIVE run (no recognized test files), emit all-zero counts. Emission is mechanical bookkeeping of the verdicts you already assigned — it introduces no new judgment, so it works in single-test mode too (the tally is just that one test's class).

Commit provenance (commitSha/dirty) — optional, for Gate's report-to-commit cross-check

Alongside the counts, run git rev-parse HEAD and git status --porcelain once and include the results as commitSha (the SHA string) and dirty (true if git status --porcelain printed anything, false otherwise) (#177, ADR-0043). This is the honest signal only you, the producer, have at audit time — Gate cross-checks commitSha against its own --commit and caps an otherwise-ship-eligible verdict at canary when they don't match (this report is evidence about a different commit than the one being gated), named in Gate's rationale, never a reason for /audit-test itself to change its verdict. Both fields are optional and purely additive: omitting them leaves the emission exactly as it behaves without this section. dirty: true is not a problem to fix before emitting — a mutation audit legitimately runs against uncommitted changes; it only means Gate discloses that no single commit fully describes this run, never a reason to cap on its own.

A certification run (--certify, Batch mode) emits the same shape with no schema change — a floor-clearing tally Gate ingests exactly as any other. Two scope-side additions keep it honest (ADR-0038): scope self-labels the run (e.g. "certify(floor=50%) · --changed"), and when the certified scope is narrower than the whole suite the emission adds a rationale line naming it ("certified scope: --changed (12 of ~180 suite test files) — ship recommendation is scoped to this changeset"), so a certified --changed ship never reads broader than it is. scope is a free-text disclosure, not a new predicate field the decision reads — but Gate does parse it as an optional passthrough and prints it verbatim in its own rendered report, next to the deep-audited fraction, so the disclosure reaches whoever reads Gate's report, not only a reader of this raw emission.

Run trace (runs[]) — optional, execution-confirmed subset only

Alongside the counts, add one runs[] record for every test a mutation was actually run against — i.e. every 🟢 confirmed-solid and 🔴 confirmed-hollow verdict from Step 4, exactly as you observed it, before you revert:

• test — the identifier you targeted (<file>::<test name>).
• mutation — the one-line change you applied (what you'd write in Proof).
• command — the exact single-test command you ran (reference/run-one-test.md), copy-pasteable, not paraphrased.
• outcome — "killed" for 🟢 (the test failed as it should), "survived" for 🔴 (the test stayed green).
• exitCode — the single-test run's actual process exit code. It must agree with the outcome: a killed record failed as it should, so its exitCode is non-zero; a survived record stayed green, so its exitCode is 0. Gate rejects a self-contradictory record (killed+exitCode:0 or survived+non-zero exit) — degrading the whole emission to opaque — so report the exit code you actually observed, not a placeholder.

Never fabricate a record. runs[] is bookkeeping of a run you already performed, not a new claim — add a record only for a test you actually mutated and ran this session, with the mutation, command, and exit code you actually observed. A 🟡 Likely verdict (env not runnable, or the mutation never reached a running app) and a ⚠️ Baseline-lock verdict get no record — they were never execution-confirmed, and a record would misrepresent reasoning-only work as proof. Unexamined tests never had a mutation proposed at all, so they never get one either. runs.length must never exceed deepAudited, and the record count for each outcome must match its count (killed records = confirmedSolid, survived records = confirmedHollow). Each record must also be a distinct (test, mutation, command) triple — a confirmed outcome maps to a distinct test identity, so don't repeat one record to pad a count. Gate cross-checks all of this and degrades an inconsistent tally to opaque rather than trusting it (ADR-0037 §3). runs[] is entirely optional: omitting it leaves the emission exactly as it behaves today.

Likewise the counts must be exhaustively accounted for: every deep-audited test lands in exactly one verdict class, so confirmedSolid + confirmedHollow + likelyHollow + baselineLock must equal deepAudited (not merely be ≤ it) — a deep audit with no recorded outcome is rejected as inconsistent.

Explain Mode (--explain)

Usage: /audit-test tests/booking.spec.ts --explain. When present, append after the standard report:

### Why This Matters
[1-2 plain-language sentences per flagged category on why this class of false confidence is
dangerous generally — e.g. why an overmocked test is worse than no test because it looks like
protection while guarding nothing. Link to GLOSSARY.md terms rather than re-explaining,
e.g. "See: False Positive Test", "See: Loose Assertion".]

The default report already teaches specifically (it names the mutation and the fix); --explain adds the generalizable concept.

Safety rule (mutations edit source)

Deep audit changes real files, so it inherits the same hard rule as prune-tests' apply mode (ADR-0001):

1. Refuse to run mutations unless git status reports a clean tree (or the user points to a scratch copy). Print the reason and fall back to 🟡 (reasoning only) otherwise.
2. Mutate → run the single test → revert — one test at a time, never leave more than one mutation live.
3. Revert immediately after each single-test run, and verify the tree is clean with git status before finishing. The clean-tree precondition makes recovery possible if a run dies mid-mutation — git checkout -- <file> restores the source — but that is a manual recovery step, not a guarantee the skill can make about a crashed session. Never leave a mutation live between tests.

Notes

• This is not coverage-review. Don't flag missing paths or propose new tests — that's the additive skill. This judges tests that already pass.
• This is not prune-tests. Don't propose deleting anything. A false-confidence test is often guarding a real behavior badly — the fix is usually to strengthen it, not remove it.
• audit-test vs Stryker. audit-test is the judgment tool — interactive, per-test, no setup, a taxonomy verdict + a concrete fix; it never produces a codebase-wide mutation score. Use it to review a PR or a suspect test and get a fix; reach for StrykerJS for periodic suite health, a defensible mutation score, or release gating. Run audit-test first (it's cheap), fix what it flags, then spend Stryker's minutes-to-hours if you need the number. (ADR-0004.)