Mutation testing
Plug-and-play skills and prompts for every AI coding agent
npx -y skills add Amey-Thakur/AI-SKILLS --skill mutation-testingAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- 18 days oldThe repository was created 18 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
- 4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Measure how strong a test suite is by mutating the code and checking the tests notice, exposing assertions that never actually fail. Use when coverage is high but you doubt the tests would catch a real regression.
SKILL.md
2.6 KB, as published. Nobody here has run it
Mutation testing
Line coverage tells you a line ran, not that a test would fail if the line
were wrong. Mutation testing closes that gap: a tool makes small changes to
the code, flips a < to <=, replaces a return with a constant, deletes a
call, then reruns the suite. A mutant the tests kill proves they guard that
behavior. A mutant that survives is a hole coverage could not see, a line
executed but never truly asserted.
Method
- Run a mutation tool for your stack. Stryker for JavaScript and C#, PIT for the JVM, mutmut or cosmic-ray for Python, cargo-mutants for Rust. Point it at the module whose tests you distrust, not the whole repo on the first pass.
- Read the mutation score as a diagnostic, not a target. The score is killed mutants over total. Chasing 100% burns effort on equivalent mutants, changes that cannot alter behavior. Treat survivors as a reading list, not a number to maximize.
- Kill survivors by strengthening assertions. A survivor usually means a
test ran the line but asserted nothing about its effect. Add the assertion
that pins the boundary the mutant moved, for instance
<=versus<at a threshold. - Recognize equivalent mutants and move on. Some mutants leave behavior identical: reordering commutative operations, a timeout the tests cannot observe. Mark them ignored with a reason rather than contorting a test to kill something untestable.
- Scope runs to survive the cost. Mutation testing reruns the suite once
per mutant and is slow. Restrict it to changed files in CI with a
--sincediff mode, or run a full sweep on a schedule against critical modules. - Gate only the code that matters. For a payment or auth core, fail the build when new survivors appear. Elsewhere, run it periodically to find rotted tests instead of blocking every commit.
Signals
- Do surviving mutants cluster on branches the team assumed were tested?
- After you kill a survivor, does the new assertion fail if you reintroduce the original bug by hand?
- Is the runtime scoped so mutation testing informs work rather than blocking it?
Boundaries
Mutation testing grades an existing suite, it does not write tests or choose what to cover: pair it with unit-test-design to fill the holes it finds. Equivalent mutants and runtime cost make it a periodic audit for critical code, not a gate on every file.