Test strategy

test-strategy

Coverage % is a number with no method. This skill makes the QA plan defensible: what to test, why that set, and a real signal that the tests bite.

qa-engineer emits docs/qa-reports/TEST-STRATEGY-{slug}.md capturing the four decisions below; the QA gate checks the file exists for medium/large features.

1. Pyramid ratio — pick by archetype (not by reflex)

The right unit:integration:e2e mix depends on where the risk lives.

Archetype	unit	integration	e2e	why
library / devtools / cli	80%	15%	5%	logic-dense, few I/O seams
web-app / saas	60%	30%	10%	request→db→render seams dominate
commerce / fintech / marketplace	50%	35%	15%	money paths need cross-component proof
data-platform / streaming	45%	45%	10%	correctness lives in pipelines, not units
ai-system / agent-product	50%	20%	10% + evals 20%	behaviour is the contract → eval set (see [[decision-eval]])

State the chosen ratio in TEST-STRATEGY and justify any deviation.

2. Case selection — equivalence / boundary / property

Don't enumerate inputs; partition them.

• Equivalence classes — one representative per class of behaviour (valid, invalid, empty, max). Testing 5 valid ids ≠ testing 5 classes.
• Boundaries — the bug lives at the edge: 0, 1, n-1, n, n+1, off-by-one, empty, overflow, the threshold itself.
• Property-based — for pure/transform logic, assert invariants over generated input (round-trip decode(encode(x))===x, idempotency, ordering) instead of hand-picked cases. Use fast-check / Hypothesis where the logic warrants it.

3. Mutation score — the only coverage that proves the tests bite

Line/branch coverage proves code ran, not that a test would fail if it broke. Mutation testing (Stryker / mutmut / cargo-mutants) flips operators/conditions and checks a test catches it. A "90% covered" module with 30% mutation score has assertion-free tests. Target: mutation score ≥ 60% on changed logic-dense files; report it in TEST-STRATEGY when the change is logic-heavy. Coverage % is the cheap proxy; mutation score is the truth.

4. Flake quarantine — a flaky test is worse than no test

A test that fails ~randomly trains the team to ignore red. Policy:

• A test that fails non-deterministically is quarantined (skipped + tracked in a Beads bug), not left to rot the signal.
• Quarantine is time-boxed: a bug with an owner and a deadline, not it.skip forever.
• Root-cause before re-enabling (shared state, timing, real network, ordering).
• This is the same discipline the eval runner needs (see the eval flaky flag in tests/eval/runner.mjs — stddev > 0.1 is a flake signal).

Output: TEST-STRATEGY-{slug}.md

# TEST-STRATEGY-{slug}
- Archetype + chosen pyramid ratio (+ justification for any deviation)
- Equivalence classes & boundaries enumerated per changed unit
- Property-based candidates (which logic, which invariants)
- Mutation-score target + result (for logic-dense changes)
- Quarantine list (flaky tests + their tracking bug)

Done = the plan answers "why THIS set of tests", not just "how much is covered".