Mutation testing

Mutation Testing

This skill makes an AI agent set up and interpret mutation testing: StrykerJS for JS/TS projects (Jest or Vitest runners) and mutmut for Python. Mutation testing injects small bugs (mutants) into source code and re-runs the tests — if the tests still pass, the mutant "survived" and your suite has a blind spot that line coverage never showed. Trigger this when a team claims high coverage but still ships regressions, when reviewing test suite quality, or when the user mentions Stryker, mutmut, or mutation score.

Core Principles

1. Line coverage measures execution, mutation score measures verification. A test that calls a function and asserts nothing gives 100% line coverage and a 0% mutation score. Mutation score is the honest metric.
2. Never mutate everything on every run. A full mutation run on a real codebase takes hours. Use incremental mode locally and scope CI runs to changed files. Full runs belong in nightly jobs.
3. A surviving mutant is either a missing assertion or dead code. Both are findings. If you cannot write a test that kills a mutant, the mutated code path is unreachable — delete it instead of suppressing the mutant.
4. Set break thresholds, not aspirational targets. Start at your current score minus 2, fail the build below it, and ratchet up monthly. A threshold nobody enforces is documentation fiction.
5. Equivalent mutants exist; budget for them. Some mutations (e.g., i < len to i <= len on an array that is never exactly full) produce identical behavior. Mark them ignored explicitly rather than chasing 100%.
6. Kill mutants with stronger assertions, not more tests. The fix for a survived > → >= mutant is usually one boundary-value assertion in an existing test, not a new test file.

Setup: StrykerJS (TypeScript + Vitest)

npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner
npx stryker init

// stryker.config.json
{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "testRunner": "vitest",
  "mutate": ["src/**/*.ts", "!src/**/*.test.ts", "!src/**/*.d.ts", "!src/generated/**"],
  "coverageAnalysis": "perTest",
  "reporters": ["html", "clear-text", "progress", "json"],
  "thresholds": { "high": 85, "low": 70, "break": 65 },
  "incremental": true,
  "incrementalFile": ".stryker-tmp/incremental.json",
  "timeoutMS": 10000,
  "concurrency": 4
}

For Jest projects swap the runner:

npm install --save-dev @stryker-mutator/core @stryker-mutator/jest-runner

// stryker.config.json (Jest variant)
{
  "testRunner": "jest",
  "jest": { "projectType": "custom", "configFile": "jest.config.js" },
  "coverageAnalysis": "perTest",
  "mutate": ["src/**/*.ts", "!src/**/*.test.ts"],
  "thresholds": { "high": 85, "low": 70, "break": 65 }
}

Run it:

npx stryker run                          # full run, writes reports/mutation/mutation.html
npx stryker run --mutate src/pricing.ts  # scope to one file while fixing survivors

Reading a Survived Mutant and Killing It

Stryker output:

[Survived] ArithmeticOperator
src/pricing.ts:14:31
-   return subtotal * (1 - discountRate);
+   return subtotal / (1 - discountRate);
Ran 3 tests, none failed.

The suite never asserts a discounted price with a nonzero rate. Kill it with a boundary assertion:

// src/pricing.test.ts
import { describe, expect, it } from 'vitest';
import { applyDiscount } from './pricing';

describe('applyDiscount', () => {
  it('multiplies subtotal by the inverse discount rate', () => {
    // 200 * (1 - 0.25) = 150; the division mutant would return 266.67
    expect(applyDiscount(200, 0.25)).toBe(150);
  });

  it('returns the subtotal unchanged at rate 0', () => {
    expect(applyDiscount(99.5, 0)).toBe(99.5);
  });
});

Ignore a genuinely equivalent mutant inline instead of lowering the threshold:

// Stryker disable next-line EqualityOperator: loop bound is equivalent for empty input
for (let i = 0; i < items.length; i++) {

Setup: mutmut (Python + pytest)

pip install mutmut pytest

# pyproject.toml
[tool.mutmut]
paths_to_mutate = ["src/"]
tests_dir = ["tests/"]
also_copy = ["conftest.py"]

mutmut run                 # mutates src/, runs pytest per mutant, caches results
mutmut results             # summary: killed / survived / timeout / suspicious
mutmut show src.pricing.apply_discount__mutmut_3   # diff of one survivor
mutmut run --max-children 4                        # parallel workers

Killing a Python survivor follows the same pattern — the mutant tells you the missing assertion:

# tests/test_pricing.py
import pytest
from src.pricing import apply_discount


def test_apply_discount_uses_multiplication():
    # mutant changed * to /; 200 * 0.75 == 150, 200 / 0.75 == 266.67
    assert apply_discount(200, 0.25) == 150


def test_apply_discount_rejects_rate_above_one():
    # kills the mutant that removed the validation guard
    with pytest.raises(ValueError):
        apply_discount(100, 1.5)

CI Integration

Incremental Stryker on pull requests, full run nightly:

# .github/workflows/mutation.yml
name: mutation
on:
  pull_request:
    paths: ['src/**', 'stryker.config.json']
  schedule:
    - cron: '0 2 * * *'

jobs:
  stryker:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # incremental mode diffs against git history
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - name: Restore incremental cache
        uses: actions/cache@v4
        with:
          path: .stryker-tmp/incremental.json
          key: stryker-incremental-${{ github.base_ref || 'main' }}
      - name: Mutation test (incremental on PRs, full nightly)
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            npx stryker run --incremental
          else
            npx stryker run --force
          fi
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: mutation-report
          path: reports/mutation/mutation.html

The thresholds.break: 65 in config makes stryker run exit non-zero below 65% — no extra scripting needed.

Best Practices

• Start mutation testing on your most critical module (pricing, auth, parsers), not the whole repo. One high-signal report beats a week-long full run.
• Use coverageAnalysis: "perTest" — Stryker then only runs the tests that cover each mutant, often a 10x speedup.
• Commit .stryker-tmp/incremental.json to CI cache, never to git.
• Track mutation score per module in the JSON report; aggregate scores hide a 30% module behind a 90% repo average.
• Set timeoutMS explicitly. Mutants that create infinite loops are killed by timeout; the default factor-based timeout misbehaves on very fast suites.
• In Python, run mutmut run against a single module first: paths_to_mutate = ["src/billing.py"].

Anti-Patterns

• Chasing 100% mutation score. Past ~90% you are writing tests for equivalent mutants. Mark them ignored with a reason and move on.
• Running full mutation on every PR. It will take 40+ minutes and the team will delete the workflow within a month. Incremental on PRs, full nightly.
• Treating "survived" as a Stryker bug. Over years of use, genuine false positives are rare; assume your suite is the problem first.
• Mutating test files or generated code. Always exclude *.test.ts, *.spec.ts, migrations, and codegen output in mutate globs.
• Lowering break to make a red build green. The threshold ratchets up, never down; fix the survivors or explicitly ignore equivalents inline.
• Writing assertion-free "kill tests" that snapshot huge objects just to kill mutants. A surviving mutant deserves a precise boundary assertion, not a snapshot blanket.

When to Trigger This Skill

• The user mentions Stryker, mutmut, mutation score, or "are my tests actually good".
• A project has high line coverage but recurring production regressions in covered code.
• Reviewing or hardening a test suite for a critical module (payments, auth, data migration).
• Setting up CI quality gates beyond line coverage thresholds.
• A stryker.config.json, stryker.conf.mjs, or [tool.mutmut] section already exists in the repo.