Validating prs with evidence

Validating PRs with Evidence

Overview

Turn a PR diff into reproducible claims: revision -> scenario -> observation -> evidence -> status.

Non-negotiable contract

1. Observe and report; do not fix code unless explicitly asked later.
2. Prove repository, revision, runtime, and environment before testing or mutation.
3. Never mutate production or an unidentified environment.
4. Pass requires executing the stated scenario; code, logs, screenshots, or similar flows alone are insufficient.
5. Record defects, then continue independent scenarios from a safe state.
6. Preserve sessions and dirty worktrees; compare start/end state without silent cleanup.
7. Use one stateful-GUI driver. Parallel work is read-only analysis only.

Workflow

Track this checklist:

PR QA
- [ ] Identify sources of truth and exact revision
- [ ] Map changed surfaces, actors, states, data, and transitions
- [ ] Baseline runtime, logs, repository, account, and environment
- [ ] Plan safe test data and cleanup/residue handling
- [ ] Execute every feasible scenario and capture evidence immediately
- [ ] Record defects, then continue independent coverage
- [ ] Recheck logs, data, repository, and artifact integrity
- [ ] Report Pass / Fail / Blocked / Not Run without conflation

Read scenario-design.md before execution and evidence-contract.md before capture or reporting. Use report-template.md by default and match the user's language.

Global blocker handoff

If a global prerequisite blocks execution, report the preflight as Blocked and mark every unexecuted scenario Not Run. Do not stop at a generic access request. Still provide:

• a surface map labeled provisional and incomplete, explicitly stating that final coverage must be rebuilt from the exact diff, acceptance criteria, and callers or consumers once access exists;
• a scenario matrix covering positive, negative, boundary, transition, repetition, actor/data/role, concurrency/idempotency, presentation, and regression dimensions when applicable, with an explicit reason for each omission;
• per-scenario evidence requirements plus the safe test-data ledger and residue plan;
• the exact repository revision and runtime identity required before execution;
• a statement that any future defect will be recorded before independent scenarios continue from a safe state;
• planned start/end reconciliation for repository, logs, data, and runtime/session;
• a commitment to open, sanitize, and link-check every final artifact;
• exact access prerequisites and the merge-confidence boundary.

A user-reported defect that was not reproduced is unverified risk, not a Fail or a confirmed fix requirement.

Scenario result contract

Record:

• ID, behavior, preconditions, and exact test data
• numbered actions and expected/actual results
• Pass, Fail, Blocked, or Not Run
• errors, evidence paths, side effects, and residue

Status	Meaning
Pass	Executed as written; all assertions observed.
Fail	Executed; at least one product assertion failed.
Blocked	Attempted, but an external prerequisite prevented the assertion.
Not Run	Deliberately not attempted; state why and retain the risk.

Stop conditions

Stop mutation when production, secrets, data loss, missing authorization, or destruction of reproducible state is possible. Otherwise continue feasible coverage.

Common mistakes

• Happy-path-only: include absence, invalidity, boundaries, transitions, repetition, roles, and stale state.
• Unavailable-data Pass: mark the precise assertion Blocked or Not Run.
• Late evidence: capture each meaningful state immediately.
• Log-noise diagnosis: correlate actions/timestamps and separate setup, product, data, environment, and unrelated observations.
• Uninspected screenshots: open, sanitize, and link-check every final image.
• Public report leakage: sanitize local paths, account details, and environment identifiers for the report's audience.
• Generic blocked handoff: retain the unexecuted scenario and evidence plan instead of returning only an access request.
• Unverified defect escalation: withhold approval if warranted, but do not call a reported symptom a demonstrated Fail before reproducing it.

Skill verification

evals/evals.json defines three cross-domain evaluations and a twelve-item scoring rubric. Run them in fresh contexts before treating behavioral changes to this skill as fully verified.