Flaky test diagnosis

Flaky test diagnosis

A flaky test is a test whose result depends on something it does not control: a random seed, another test's leftovers, the wall clock, or thread timing. The trap is re-running until it passes and calling it fixed, which just hides the same nondeterminism from the next person. The job is to seize the hidden variable until the failure becomes reliable, then remove it.

Method

1. Reproduce the failure in a tight loop first. Run the single test hundreds of times: pytest --count=500 path::test, go test -run X -count=500, or a shell loop until it fails. A flake you can trigger 1 in 50 locally is debuggable; one you only see in CI needs the environment pinned before anything else.
2. Pin the random seed and read what it was. Frameworks randomize: capture and print the seed on failure, then replay it. pytest -p randomly --randomly-seed=<n>, a fixed random.seed(n), or Random(n) in the test. A failure that reproduces under one seed is a data-dependent bug, not cosmic flakiness.
3. Shuffle and isolate test order. Order-dependent flakes come from shared state one test leaves for another: a mutated global, an unclosed connection, a row not rolled back. Run the suite shuffled (pytest-randomly, go test -shuffle=on) to expose it, then run the suspect test alone; if it passes alone but fails in the suite, hunt the leaked state.
4. Freeze the clock and time zone. Tests that read now() fail at midnight, month ends, or on a slow machine where a timeout expires. Inject a fixed clock (freezegun, libfaketime, Clock injection) and set TZ. Replace real sleep-and-poll waits with deterministic waits on a condition, never a bare timeout tuned to a fast laptop.
5. Quarantine external and timing dependencies. Network calls, real filesystems, and unordered concurrency each add nondeterminism. Stub the network, use a temp dir per test, and await completion rather than sleeping. If concurrency is the source, the race-conditions skill applies to the code under test, not the test itself.
6. Prove it with the same loop, then re-enable randomness. The loop from step 1 must pass a long run with the fix in place. Only then remove the pinned seed and fixed order so the test guards real variation again. A test kept green only by freezing every input has stopped testing.

Litmus tests

• Under one captured seed and fixed order, does the failure reproduce every run?
• Does the test pass alone and fail in the suite, or fail both ways?
• After the fix, does a 500-iteration shuffled loop stay green?

Boundaries

Some flakiness is the code under test being genuinely nondeterministic (unsynchronized threads, ordering assumptions on a hash map); fix the code, not the test. Skipping or retrying a flake is a stopgap that hides real races, and it belongs behind a tracked ticket, not in the merged suite.