Detox

Detox

A flaky test is worse than a missing test: it trains everyone to ignore red. Your job is to separate flaky from broken, diagnose each flake's root cause class, and fix the cause — never the symptom.

Banned fixes

Retry wrappers, longer sleeps, and skip are not fixes; they are how flakes become permanent residents. The only acceptable retry is around a call to a genuinely external system the test cannot control — and that usually means the test should be mocking it instead.

Phase 1 — Census

Run the full suite repeatedly — default 5 runs, more if runs are cheap — and build a per-test pass/fail matrix.

• Fails every run → broken, not flaky. Report separately; out of scope unless asked.
• Fails some runs → flaky. This is the target list, ranked by failure rate.
• Record conditions per run: ordering, parallelism, seed, timing — the variance between a passing and failing run often names the culprit by itself.

Phase 2 — Diagnose each flake

Isolate, then classify. Standard experiments:

Experiment	If behavior changes →
Run the test alone vs. after the full suite	Shared state / test-order dependence
Run the suite in random order / different parallelism	Order dependence, resource contention (ports, files, DB rows)
Run with a fixed seed / frozen clock	Unseeded randomness, time-of-day or timezone dependence
Run repeatedly in a tight loop	Race conditions, timing assumptions, hardcoded timeouts
Run with network blocked / observed	Real network calls hiding in a "unit" test

Read the failing assertion under each hypothesis: a flake almost always belongs to one of six families — shared state, ordering, timing/races, unseeded randomness, clock/timezone, external systems (network, filesystem, ports).

Phase 3 — Fix the cause

Match the fix to the family: proper setup/teardown isolation for shared state; explicit awaiting or synchronization points instead of sleeps for races; injected/frozen clocks; seeded randomness; mocks or test doubles for external systems; per-test unique resources (temp dirs, ephemeral ports) for contention. The fix should make the test deterministic, not tolerant.

Phase 4 — Prove it

A fix isn't proven by one green run. Run the fixed test 20+ times, including under the conditions that made it fail (full-suite ordering, parallelism, tight loop). Then run the whole suite to confirm the fix didn't shift the flake elsewhere.

Phase 5 — Report

Write FLAKES.md: per test — failure rate before, root-cause family, the experiment that proved it, the fix, runs-passed after. Plus a short "flake debt" section: patterns in the codebase likely to breed the same families again (a shared fixture, a real HTTP client in unit tests), so the next flake gets prevented rather than hunted.