Mutation testing

Mutation Testing

Tests pass ≠ tests are useful. This skill uses Claude to generate meaningful mutations for source code, then runs the project's existing test suite against each one. A surviving mutant is a gap — a real bug the suite would miss.

What this skill does

1. Discovers source files — the calling agent reads the project structure and selects files
2. Detects test runner — the calling agent inspects project config and builds the test command
3. Checks baseline — verifies the test suite passes before mutating anything
4. Generates mutations via subagents — dispatches one Task per source file in parallel; each subagent reads the file and returns a JSON array of mutations; results are merged into a single mutations.json file
5. Applies each mutation — writes the mutated file to disk, one at a time
6. Runs the test suite — records Killed (tests failed) vs. Survived (tests passed)
7. Restores each file — always restores the original before the next mutation
8. Generates mutation-todos.md — table of survivors with rationale for each gap
9. Persists to logbook — SQLite + JSONL under ./.logbooks/mutation-testing/
10. Warns if score < 70% — exits with code 2

---

How to invoke this skill

This skill follows a two-phase protocol: generate then run.

Phase 1 — Generate mutations (caller's responsibility)

Before dispatching subagents, the calling agent must:

1. Discover source files by reading the project structure. Prefer src/, lib/ directories; skip test files (test_*.py, *_test.py, *.spec.*, *.test.*, paths with test/ or tests/), and skip non-source directories (.venv, .git, dist, build, node_modules, __pycache__, site-packages). Cap at 20 files.

2. Detect the test runner by inspecting the project root, then build the full test command:

   Signal	Command
   pytest.ini, conftest.py, or [tool.pytest] in pyproject.toml	python -m pytest --tb=no -q --no-header
   jest.config.* or "jest" in package.json devDependencies	npx jest --passWithNoTests --forceExit
   "vitest" in package.json devDependencies	npx vitest run
   "mocha" in package.json devDependencies (no babel)	npx mocha --recursive
   "mocha" + "@babel/register" in package.json devDependencies	npx mocha --require @babel/register --recursive
   Any *.go file present	go test ./...
   "test" script in package.json	npm test

   Pass the full command via --test-command in Phase 2.

Then, for each source file to mutate, dispatch a subagent with this prompt template:

You are a mutation testing expert. Generate up to {max_per_file} meaningful mutations for the following source file that will expose gaps in the test suite.

Already-known open gaps (do NOT regenerate these — find new ones): {known_gaps_for_file}

File: {rel_path}

{line_numbered_source}

Return a JSON array. Each element must be an object with these keys:

• id (string) — unique ID, e.g. "m1"
• mutatorName (string) — operator name, e.g. "FlipGreaterThan"
• line (integer) — 1-based line number
• original_line (string) — complete original line, leading whitespace preserved
• mutated_line (string) — complete mutated line, leading whitespace preserved
• replacement (string) — short change description, e.g. "> → >="
• rationale (string) — why this exposes a test gap
• file (string) — use exactly the path from the "File:" line above, with forward slashes

Target: conditional flips (> → >=), boolean negation, arithmetic swaps, return value changes, string literal blanking, boundary shifts, argument removal/swap. Skip: comments, imports, type annotations, docstrings, blank lines, logging.

Collect all subagent outputs, merge into a single JSON array, and save to a temp file (e.g. /tmp/mutations.json).

Tips:

• Dispatch subagents in parallel (one per file) for speed
• Pass --model claude-sonnet-4-6 as a logbook hint if using a cheaper model
• Query prior open gaps first (from the logbook SQLite) and pass them per-file so subagents generate novel mutations rather than re-discovering known gaps

Phase 2 — Run the script

python plugins/mutation-testing/skills/mutation-testing/scripts/mutation_testing.py \
  --mutations-file /tmp/mutations.json \
  --test-command python -m pytest --tb=no -q

With options:

# Jest project
python ... --mutations-file /tmp/mutations.json \
  --test-command npx jest --passWithNoTests --forceExit

# Custom threshold and timeout
python ... --mutations-file /tmp/mutations.json --threshold 80 --timeout 120 \
  --test-command python -m pytest --tb=no -q

# Record which model was used for generation (metadata only)
python ... --mutations-file /tmp/mutations.json --model claude-sonnet-4-6 \
  --test-command python -m pytest --tb=no -q

# Skip logbook writes
python ... --mutations-file /tmp/mutations.json --no-logbook \
  --test-command python -m pytest --tb=no -q

Note: --test-command must be the last flag — all remaining arguments after it are captured as the test command.

CLI flags

Flag	Default	Description
--mutations-file PATH	required	JSON file of pre-generated mutations
--test-command WORD …	required	Full test command (must be last flag)
--threshold	70	Minimum acceptable mutation score
--timeout	60	Seconds allowed per test run
--model	none	Claude model used for generation — recorded in logbook as metadata (optional)
--skip-baseline	off	Skip the pre-mutation baseline pass check
--no-logbook	off	Skip SQLite/JSONL logbook writes

Exit codes

Code	Meaning
0	Success — score ≥ threshold
1	Setup, load, or baseline failure; or all mutations skipped
2	Score below threshold

---

Prerequisites

• Python 3.10+ (uses match/case and str.removesuffix)
• A working test suite (tests must pass before running mutation tests)
• At least one source file (discovered by the calling agent)
• No SDK or API key required — mutation generation is done by the calling Claude agent

---

Mutant statuses

Status	Meaning
Killed	Tests detected the bug — good
Survived	No test caught the bug — test gap
Error	Test runner failed or errored
Skipped	Original line didn't match the expected text

Score formula

score = Killed / (Killed + Survived + Errors) × 100

Skipped mutants are excluded from the denominator. If all mutants are skipped, the script exits 1 (setup issue, not a score failure).

Score	Assessment
≥ 90%	Strong test suite
70–90%	Acceptable — review survivors
< 70%	Weak — prioritise mutation-todos.md

---

Output artifacts

Artifact	Location	Description
mutation-todos.md	repo root	Survivors table with rationale
SQLite ledger	./.logbooks/mutation-testing/{slug}.sqlite	Run history + survivors
JSONL trace	./.logbooks/mutation-testing/{slug}.jsonl	Append-only run log

---

Logbook spec

Full schema, query examples, and export setup: mutation-runs.logbook.md.