Flaky test reduction

Flaky Test Reduction

Instructions

Flake is not a test problem — it is usually a design problem in either the test or the code under test. Retrying a flaky test without understanding why makes the suite untrustworthy and slow. This skill is a root-cause checklist and a set of fixes.

1. First Principle: Isolate the Cause

A test is flaky for exactly one of a handful of reasons. Work the list in order; do not skip:

1. Time (clocks, timeouts, delays).
2. Concurrency (race conditions, main-thread assumptions).
3. Shared state (singletons, disk, DB, DI graph reused).
4. Ordering (test depends on another test's side effects).
5. External dependency (network, device service, third-party SDK).
6. Resource pressure (OOM, CPU-starved emulator).
7. Framework bug (rare; only conclude this last).

2. Kill Time-Based Flake

• Replace System.currentTimeMillis(), Date(), DateTime.now(), Date.now() with an injected clock.
• Replace Thread.sleep, Task.sleep, Future.delayed, setTimeout in tests with condition-based waits.
• Use virtual time for retry/backoff logic:

@Test fun retries_with_backoff() = runTest {
    val dispatcher = StandardTestDispatcher(testScheduler)
    val sut = Sync(dispatcher = dispatcher, clock = FixedClock(T0))
    sut.start()
    testScheduler.advanceTimeBy(5_000) // virtual
    assertEquals(3, sut.attempts)
}

3. Kill Concurrency Flake

• Use test dispatchers / schedulers (TestDispatcher, TestScheduler, fakeAsync, jest.useFakeTimers).
• Never assert on state right after launch { } / Task { } / then() without joining or awaiting.
• In UI tests, use the framework's idling primitive (IdlingResource, waitForExistence, pumpAndSettle, waitFor(...) in Detox).
• Beware "it works because my laptop is fast" — run the suite under CPU throttling occasionally to surface races.

4. Kill Shared-State Flake

• Build and tear down per test: DB (in-memory), HTTP server, DI container.
• Purge global caches in @AfterEach / tearDown.
• On Android, use @After to reset Dispatchers.setMain / Dispatchers.resetMain.
• On iOS, reset UserDefaults test suite via removePersistentDomain(forName:).
• Avoid object/singleton collaborators in production code — inject them so tests get fresh instances.

5. Kill Ordering Flake

Run tests in random order in CI. Most runners support it (-Dspock.configuration.runner.reverseOrder, JUnit5 random, XCTest -randomize-tests, Jest --testSequencer). A test that fails only in some order is leaking state; fix the leak, not the order.

6. Kill External-Dependency Flake

Any dependency on a live backend or third-party SDK will flake. Stub them:

• Run E2E against a dedicated seeded environment, not shared staging.
• Replace analytics, push, and crash SDKs with no-op fakes in tests.
• Pin time-sensitive flows so they do not depend on real wall-clock (holidays, weekends, token expiry).

7. Kill Resource-Pressure Flake

• Reduce parallelism on under-powered CI runners.
• Cap emulator animation and transition duration (adb shell settings put global window_animation_scale 0).
• Pre-boot simulators / emulators before the test phase starts so boot cost is not inside a test timeout.
• Raise device-farm test timeouts proportionally to device class.

8. Retries — Use Cautiously

Retries hide flake, they do not fix it. Use them only as a bridge:

• Retry once at the suite level; report the retry count as a metric.
• Quarantine any test that needs retry to pass more than twice in a week.
• Never retry inside the test body itself — it masks the root cause.

9. Quarantine and Budget

• Move flaky tests to a @Flaky / @Tag("flaky") group that still runs but does not fail the build.
• Set a time box: 30 days to fix or delete.
• Track flake rate per test; surface the top 10 in a weekly report.

10. Tooling

• Kotlin: kotlinx-coroutines-test, MockWebServer, runTest.
• Swift: XCTWaiter, XCTNSPredicateExpectation, Task.yield() when awaiting a state flip.
• Flutter: fakeAsync, tester.pumpAndSettle(Duration(seconds: 5)) with explicit timeout.
• RN: jest.useFakeTimers({ advanceTimers: true }), @testing-library/react-native's waitFor.

11. Checklist

• No real clock, RNG, or wall-clock sleep in tests.
• Every async assertion is gated by a condition wait, not a fixed delay.
• Test suite runs in random order in CI.
• DB, HTTP server, and DI graph are rebuilt per test.
• External SDKs are replaced with fakes.
• Retries are a bridge, not a fix; retry counts are tracked.
• Quarantine has a 30-day fix-or-delete rule.
• Top-10 flakiest tests are published weekly.