Test strategy
Don't buy software. Get the work done. GreatCTO ships AI autopilots that run a whole business function — medical coding, legal docs, procurement, accounting, IT, tax — from intake to outcome. A qualified human signs only the judgment calls. Live connectors, built-in compliance.
npx -y skills add avelikiy/great_cto --skill test-strategyAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
What its author says it does
Copied from the file, not written here
Coverage-design method for qa-engineer — pyramid ratios per archetype, equivalence/boundary/property case selection, mutation score as the real coverage signal, and a flake-quarantine policy. Turns "coverage is 90%" (a number with no method) into a defensible test plan. Emits TEST-STRATEGY-{slug}.md, which the QA gate checks exists.
SKILL.md
4.0 KB, as published. Nobody here has run it
test-strategy
Coverage % is a number with no method. This skill makes the QA plan defensible: what to test, why that set, and a real signal that the tests bite.
qa-engineer emits docs/qa-reports/TEST-STRATEGY-{slug}.md capturing the four
decisions below; the QA gate checks the file exists for medium/large features.
1. Pyramid ratio — pick by archetype (not by reflex)
The right unit:integration:e2e mix depends on where the risk lives.
| Archetype | unit | integration | e2e | why |
|---|---|---|---|---|
| library / devtools / cli | 80% | 15% | 5% | logic-dense, few I/O seams |
| web-app / saas | 60% | 30% | 10% | request→db→render seams dominate |
| commerce / fintech / marketplace | 50% | 35% | 15% | money paths need cross-component proof |
| data-platform / streaming | 45% | 45% | 10% | correctness lives in pipelines, not units |
| ai-system / agent-product | 50% | 20% | 10% + evals 20% | behaviour is the contract → eval set (see [[decision-eval]]) |
State the chosen ratio in TEST-STRATEGY and justify any deviation.
2. Case selection — equivalence / boundary / property
Don't enumerate inputs; partition them.
- Equivalence classes — one representative per class of behaviour (valid, invalid, empty, max). Testing 5 valid ids ≠ testing 5 classes.
- Boundaries — the bug lives at the edge: 0, 1, n-1, n, n+1, off-by-one, empty, overflow, the threshold itself.
- Property-based — for pure/transform logic, assert invariants over generated input (round-trip
decode(encode(x))===x, idempotency, ordering) instead of hand-picked cases. Use fast-check / Hypothesis where the logic warrants it.
3. Mutation score — the only coverage that proves the tests bite
Line/branch coverage proves code ran, not that a test would fail if it broke. Mutation testing (Stryker / mutmut / cargo-mutants) flips operators/conditions and checks a test catches it. A "90% covered" module with 30% mutation score has assertion-free tests. Target: mutation score ≥ 60% on changed logic-dense files; report it in TEST-STRATEGY when the change is logic-heavy. Coverage % is the cheap proxy; mutation score is the truth.
4. Flake quarantine — a flaky test is worse than no test
A test that fails ~randomly trains the team to ignore red. Policy:
- A test that fails non-deterministically is quarantined (skipped + tracked in a Beads bug), not left to rot the signal.
- Quarantine is time-boxed: a bug with an owner and a deadline, not
it.skipforever. - Root-cause before re-enabling (shared state, timing, real network, ordering).
- This is the same discipline the eval runner needs (see the eval
flakyflag intests/eval/runner.mjs— stddev > 0.1 is a flake signal).
Output: TEST-STRATEGY-{slug}.md
# TEST-STRATEGY-{slug}
- Archetype + chosen pyramid ratio (+ justification for any deviation)
- Equivalence classes & boundaries enumerated per changed unit
- Property-based candidates (which logic, which invariants)
- Mutation-score target + result (for logic-dense changes)
- Quarantine list (flaky tests + their tracking bug)
Done = the plan answers "why THIS set of tests", not just "how much is covered".