Pitfall verification

Pitfall verification

Use this skill after finishing any PRD, spec, plan, or code artifact — before declaring the work done. It is NOT a generic review. It is a targeted check that typical pitfalls for this type of artifact, in this domain, do not apply here.

Invoke with: /superpowers-gstack:pitfall-verification

This skill is a multi-model orchestrator

For anything beyond a trivial change, one model is not enough — there is always more to find, and a single training distribution has fixed blind spots. So this skill does not stop at Claude's own pitfall pass. For substantive / ship-worthy changes it runs, as ONE automatic flow:

1. Self-pitfall (Claude, lens 1) — the rounds below, max 2.
2. Codex (lens 2) — /codex review on the patched artifact.
3. Third model house (lens 3) — /superpowers-gstack:third-lens-review on the patched artifact.
4. Adversarial synthesis — combine all findings, finding-is-real-until-refuted.

Stages 2–4 run automatically. Do NOT stop after the self-pitfall rounds to ask the user whether to run Codex or the third lens — run them. The whole point is that there is nothing extra to remember: invoking pitfall-verification on substantive work is the multi-model review. The only thing that skips stages 2–4 is the trivial tier (see "Tier gate" below) — a typo or doc fix gets the free self-pitfall pass and nothing else.

When to invoke

Automatically after completing:

• A PRD, spec, or design document
• An implementation plan
• A code change (feature, refactor, bug fix)
• Output from writing-specs, writing-plans, executing-plans, verification-before-completion, or any GStack planning/review skill

Run twice max. Two rounds catch almost everything; a third round has sharply diminishing returns.

Sequence

1. Self-check first (~30 seconds): placeholders, scope drift, internal consistency, ambiguity. This is the standard sanity pass. Fix anything obvious.
2. Pitfall verification (this skill): targeted check for the pitfalls below, adapted to the artifact type and domain.

Do not skip step 1 — self-check and pitfall verification are different lenses.

Pitfall lists per artifact type

These are starting points, not exhaustive checklists. Always ask: what typically goes wrong with this kind of artifact, in this specific domain? Infer additional pitfalls from the actual code/spec under review.

PRD

• Unclear user story or missing success metric
• Hidden stakeholder assumptions (who signs off? who blocks?)
• Unspecified non-functional requirements (perf, privacy, accessibility, i18n)
• Conflicting requirements across sections
• Missing failure modes (what if X is unavailable? what if load spikes?)
• Over-specification of solution vs under-specification of problem

Spec / plan

• Ambiguous contracts — input/output types, error shapes, null semantics
• Uspecified edge cases (empty, max-size, concurrent, partial failure)
• Missing error states / recovery paths
• Hidden assumptions about call order, transactions, idempotency
• Signature drift: spec references functions/fields that do not exist in the code
• Assumed external API behavior that has not been verified

Code

• Security: prompt injection, input sanitation, credential leakage, SSRF, path traversal, auth bypass
• Idempotency: side effects on retry, hidden state, ordering dependencies
• Integration contracts: field names, types, and signatures that actually match the code they call into (cross-check — do not assume)
• Edge cases: empty input, oversized input, Unicode, currencies, time zones, DST, negative numbers, leap years
• LLM output: markdown-wrapped JSON, unexpected JSON structure, hallucinated fields, schema drift
• Concurrency: race conditions, deadlocks, shared mutable state
• Resource lifecycle: leaked handles, unclosed connections, memory growth under load

How to run the check

For each pitfall on the relevant list:

1. State the pitfall — one sentence.
2. Locate the risk surface — which function, field, section, or claim could be affected?
3. Verify, do not assume. Read the actual code/spec. Cross-check field names against the implementation. Test the edge case mentally with real values.
4. Report: Not applicable (with reason) / Applicable and handled (point to where) / Applicable and not handled (propose fix).

If a pitfall is not applicable to this domain, say so explicitly — do not silently skip it. Stating "N/A because this code never touches dates" is itself a verification signal.

Domain inference

The lists above are generic-LLM-common. Real pitfalls are often domain-specific. Before the round, spend 15 seconds asking:

• What kind of system is this? (auth? payments? ETL? LLM pipeline? UI? infra?)
• What categories of bugs hit this kind of system most often?
• What did past bugs in this codebase/team look like? (check git log, CHANGELOG, incident notes if accessible)

Add those inferred pitfalls to the round before running.

Output format

End the round with a compact verdict:

Pitfall verification (round N/2):
- [pitfall] → N/A | handled at file:line | NOT HANDLED — proposed fix
- ...

Verdict: CLEAN | ISSUES FOUND (see above)

If round 1 surfaces issues, fix them, then run round 2 on the patched artifact. If round 2 is clean, declare done. If round 2 still finds issues, surface them to the user — do not silently run round 3.

Tier gate — which lenses run automatically

After the self-pitfall rounds, classify the change and run the rest of the chain automatically for its tier. Do not ask the user which lenses to run — the tier decides.

Tier	What it is	Lenses (all automatic)
Trivial	docs, typo, comment-only, test-only-coverage, WIP checkpoint	Self-pitfall only — stop here
Ship-worthy	bumps a version file, produces a CHANGELOG entry, feat/fix/refactor affecting runtime, or changes public contracts	Self-pitfall → Codex → synthesis
+ High-stakes	a ship-worthy change that also touches architecture / real-time / security / public contracts / migration logic	…→ third model house → synthesis

In practice most substantive work is at least ship-worthy, so Codex runs by default — you no longer invoke it by hand. The third house adds itself on the high-stakes subset. Both fire without a confirmation prompt; cost is reported after each call, not gated before it.

Cost-ledger — adaptive lens routing (when enabled)

The plugin ships an optional adaptive lens-router (scripts/cost-ledger/) that learns, per (domain, tier), which external lenses have stopped producing findings that survive synthesis, and safely skips them to save cost. It is safe by construction: it skips nothing until a domain has ≥10 clean reviews (cold-start), never touches the self-pitfall floor or high-blast domains (RT-audio / DSP / concurrency / migration / auth / security), and auto-reverts + quarantines any skip a shadow run shows was premature. Disable anytime by running python3 "$LEDGER" pause (the LEDGER path is defined below; status, reset, and explain <domain> work the same way — there is no /cost-ledger slash command).

Wire it at three moments. LEDGER=<this skill's base directory>/../../scripts/cost-ledger/cli.py. The ledger is advisory, never a blocker — if any call errors or cli.py is absent, log it and proceed with the FULL lens set (it only ever removes work, so absence/failure fails safe toward more verification).

1. Before dispatching external lenses (point 1). Classify the change's domain (the v0.2 classifier label) and tier, then python3 "$LEDGER" gate <domain> <tier> → JSON {"skip":[...],"shadow":[...]}. For each lens in skip, do NOT gate on it (treat as passed) UNLESS it is also in shadow — then run it this review but mark its result non-gating (a measurement, not a gate). self-pitfall is never in skip.
2. After Stage-4 synthesis (point 2). For each lens that ran (real or shadow): echo '<json>' | python3 "$LEDGER" record - with {ts, review_id, lens, domain, tier, cost_usd, findings, max_severity, survived_synthesis, shadow}. survived_synthesis = that lens's findings that survived the adversarial synthesis. (Pipe the JSON on stdin via -; do not pass it as an argv string — a large findings payload could hit ARG_MAX.)
3. Immediately after the records (point 3). echo '<json-array>' | python3 "$LEDGER" tune - with the same records — updates routing under a lock, auto-reverts escaped skips, and is a clean no-op if it can't get the lock.

Stages 2–4 — external lenses + synthesis (automatic per tier)

Run on the patched artifact, in order — each later lens reads a cleaner surface.

Stage 2 — Codex (ship-worthy and above)

Invoke /codex review on the patched artifact. Codex catches cross-file drift, concurrency contracts, and concrete run bugs (false timeouts, stale async-resume, double-acquire leaks) that self-review systematically misses. Fix what it finds. Run it automatically — do not ask first.

Idempotency guard: if /codex review has already been run on this exact patched artifact earlier in the current flow (e.g. an orchestrator like autoimplement runs /review + /codex review itself around this skill), do not re-run it — fold the existing Codex findings into the Stage 4 synthesis instead. The goal is one Codex pass per patched state, not one per skill that mentions Codex.

Stage 3 — third model house (high-stakes only)

Invoke /superpowers-gstack:third-lens-review on the patched artifact. A different model house (different training distribution → different blind spots) finds architecture-level mistakes ("you never wired it together"), degraded-state bugs, and challenged assumptions the two Western houses both took for granted. It picks the model by artifact type (--role architecture/correctness/countersynthesis; the third house runs on non-Western infra, so keep sensitive artifacts — auth/keys/health/finance — to the self + Codex lenses) and runs its own adversarial synthesis of the third-house output. Run it automatically for the high-stakes tier — do not ask first.

Stage 4 — combined adversarial synthesis (mandatory whenever stage 2 or 3 ran)

Do not dump raw findings. Fold Codex + third-house results into one verdict, adversarially: an external finding is REAL until you explicitly refute it with a reason (this counters LLM-judge agreement bias — failure-detection drops to ~50% when a judge is conciliatory, worse when it is partly judging its own earlier work). Log each dropped finding with why. Treat cross-model disagreement as the signal, not noise — every disagreement ends in an explicit, reasoned decision. Cross-model agreement = high-confidence green, no action.

Multi-lens verdict (tier: <trivial|ship-worthy|high-stakes>):
- Lenses run: self-pitfall [+ Codex] [+ <third-house model id>]
- CONFIRMED (fix now): [P1/P2] <finding> — <file:line> → <fix> (survived refutation because …)
- DISAGREEMENT → DECISION: <finding> → <explicit reasoned call>
- DROPPED (with reason): <finding> → <over-strict | handled at file:line | wrong>
- Cost: $<sum from lens footers>
- Verdict: CLEAN | FIX-THEN-RECHECK | SURFACE-TO-USER

Why two rounds

One round catches the obvious pitfalls. Round 2, run on the patched artifact, catches pitfalls that the fixes from round 1 introduced or exposed. Beyond that, returns drop fast and reviewer fatigue introduces noise.

What this skill is NOT

• Not a security audit (narrower, adversarial, deeper)
• Not a code review of style or readability
• Not a test suite (does not execute code)
• Not a replacement for verification-before-completion (which checks claims vs reality for already-finished work — pitfall verification is upstream of that)

Use it as the last check before handing off — after implementation, before the user sees "done".